MongoDB

[Andy_Pagin]

Anyone got any experience of Mongo, or non-relational databases in general?
____________________
They're coming to take me away, ho-ho, hee-hee, ha-haaa, hey-hey,
the men in white coats are coming to take me away.
Yamaha Vity -> YBR125 -> FZS600 Fazer -> FZ1-S Fazer

[Taught2BCauti...]

I have a little experience with Mongo and other 'NoSQL' types - but mostly in extracting the data from them so they could be binned.
____________________
Honda Varadero XL125(V8)
www.TheFutureIsHere.eu

[Dalemac]

Yes.

Next question?
____________________
YBR125 -> GPZ500S -> SL1000

[The Shaggy D.A.]

I spent a lot of the 80's and 90's working with Pick, in its various guises.

[edit] Also spent an unfortunate period with MUMPS.
____________________
Chances are quite high you are not in my Monkeysphere, and I don't care about you. Don't take it personally.
Currently : Royal Enfield 350 Meteor
Previously : CB100N > CB250RS > XJ900F > GT550 > GPZ750R/1000RX > AJS M16 > R100RT > Bullet 500 > CB500 > LS650P > Bullet Electra X & YBR125 > Bullet 350 "Superstar" & YBR125 Custom > Royal Enfield Classic 500 Despatch Limited Edition (28 of 200) & CB Two-Fifty Nighthawk > ER5

[Andy_Pagin]

[Rogerborg]

[Andy_Pagin]

I'm not actually management, but since I'm the programmer, analyst, business analyst and dba I'm a manager in all but name. Ultimately I make the decisions on the project.

Requirement is a system to store insurance data for the purpose of reporting and processing the recovery of re-insurance and the management of claims and reserves. This is in the run-off business, so off the shelf applications won't do, it's too specialised.
Insurance data isn't terribliy logical and data requirements for direct inwards, assumed Ri, and all the hideously complex reinsurance models vary considerably.
For example our policy datafiles contain all the policies for the client, both inwards and outwards, but individual policy records require different mandatory elements depending on the policy type. So the policy file is a bit of a mess, 75% of the fields are blank in most cases.
So I'm thinking a MongoDB would allow me to create records tailored to specific policies, and the ability to create one-to-many relationships within a single datafile, specific to individual policies would be useful.

tl;dr; I need to store loads of complex, often illogical data, much of it dating back to the 1950s and produce lots of complex reports, frequently, and very quickly.
____________________
They're coming to take me away, ho-ho, hee-hee, ha-haaa, hey-hey,
the men in white coats are coming to take me away.
Yamaha Vity -> YBR125 -> FZS600 Fazer -> FZ1-S Fazer

[Taught2BCauti...]

It's not easy to compare Mongo to SQL, because they are good at different things. Mongo is good for storing and retrieving documents, and is used as the backend system by sites like eBay, LinkedIn and CraigsList, where table-based systems wouldn't perform so well.

For your scenario, you could perhaps consider using Mongo to store all of the policy documents, and a SQL database to store policyholder data, and cross reference which documents apply to which policy.

One alternative, might be to store the documents as PDF's in a BLOB field in SQL, but that would limit search functions, if you needed to search for phrases in the documents.
____________________
Honda Varadero XL125(V8)
www.TheFutureIsHere.eu

[duhawkz]

Postgres can aslo do document databases and it out performs mongodb.
https://www.enterprisedb.com/postgres-plus-edb-blog/marc-linster/postgres-outperforms-mongodb-and-ushers-new-developer-reality

We use mongodb and MarkLogic, Marklogic is expensive and clunky, we haven't been running our own mongodb servers long enough to find its faults. Lots of things seem like and after thought with mongodb.

here are the limits for mongo
https://docs.mongodb.org/manual/reference/limits/

Nosql allows devs to quickly get apps running with out having to think about how its data is structured in the DB, instead of spending months designing databases. So devs love it because projects get moving quickly.

It also means mistake can be made, find that some of the data you put in nosql dbs, really was relational , which is not good when your months into a project?
____________________
"The guy is a worthless cunt and I honestly believe I would be a slightly happier person if he died." - Chris-Red

[ScaredyCat]

Have you taken a look at elasticsearch ? Easy to cluster and add nodes, very powerful indeed.
____________________
Honda CBF125 ➝ NC700X
Honda CBF125 ↳ Speed Triple

[Rogerborg]

elasticsearch is splendid for what it does, and can be extended fairly easily, but it's not really intended for analysis.

Mongodb is still very much try-it-and-see. Historically - and I stress that - it treated data integrity as a "yeah, we'll give it a try" feature. I'm sure it's solved that by now, it has some very good people working on it, but the thing about SQL solutions is that they've had much longer to solve for quality and performance.

Nobody ever got sacked for using PostgreSQL, is what I'm saying.

My solution would be to sack Andy and use the money to buy a SAS Visual Analytics managed solution. That might not be the answer that's being sought though.
____________________
Biking is 1/20th as dangerous as horse riding.
GONE: HN125-8, LF-250B, GPz 305, GPZ 500S, Burgman 400 // RIDING: F650GS (800 twin), Royal Enfield Bullet Electra 500 AVL, Ninja 250R because racebike

[Andy_Pagin]

Well as an analyst I of course have to consider the benefits to the business of sacking me.
Unfortunately run-off is so specialised, building anything without the help of one of the countries handful of experts (me) would be prohibitively expensive. Also since the requirements of the clients change with the frequencey and unpredicability of the weather an in-house developer and solution is the only viable way forward.
The above isn't sarcasm btw, several times over the years it's been seriously considered and rejected.
____________________
They're coming to take me away, ho-ho, hee-hee, ha-haaa, hey-hey,
the men in white coats are coming to take me away.
Yamaha Vity -> YBR125 -> FZS600 Fazer -> FZ1-S Fazer

[Rogerborg]

I holla, I picture you insurance guys as falling into a black hole of speciality, occasionally spunking off nummy nuggets of useful information as you spiral towards the event horizon where nobody else can understand a word you say.
____________________
Biking is 1/20th as dangerous as horse riding.
GONE: HN125-8, LF-250B, GPz 305, GPZ 500S, Burgman 400 // RIDING: F650GS (800 twin), Royal Enfield Bullet Electra 500 AVL, Ninja 250R because racebike

[Andy_Pagin]

[Rogerborg]

[el_oso]

good nips
____________________
Duke 390
Previous: '05 XR125L | '96 XJ600S Diversion |'05 Suzuki GSXR1000 | '05 Honda CBR125-R | '97 YZF 600R Thundercat | '11 Honda CBR250
Car: Jeep Wrangler 4.0L

[chrisw]

[Frost]

The only real benefit that document databases have over relation is the schema-less design and the ability to scale across multiple machines. Unless you're scaling to the extent of exceeding what a single machine can do i'd advise going with a relational database.

We use Marklogic for our main data source and we are experiencing a world of pain. Our needs have changed over time and the system design is needing to pull lots of data from multiple documents. This is causing unreliability due to us having to run loads of clusters, costing us a fortune (£30k per week....) and now all our teams are having to write their own service layers to take load of the database.

Relational databases are amazing in how fast they can be, along with how easy they are to understand. Migrations can be a pain though.

Document databases are getting a lot of good press as they solve some issues relational databases have always struggled with. This does not however make them a replacement. Unless you see something on the horizon that you can't do with relational, go relational.