|
Thread Rules
1. This is not a "do my homework for me" thread. If you have specific questions, ask, but don't post an assignment or homework problem and expect an exact solution.
2. No recruiting for your cockamamie projects (you won't replace facebook with 3 dudes you found on the internet and $20)
3. If you can't articulate why a language is bad, don't start slinging shit about it. Just remember that nothing is worse than making CSS IE6 compatible.
4. Use [code] tags to format code blocks. |
On May 23 2015 21:53 Manit0u wrote:Show nested quote +On May 23 2015 19:27 supereddie wrote:
I'd just recommend using a differente type of database. Like a Document database. That way jou can store the JSON as... json. Easy. No mapping required. Yeah. You can do that in SQL by simply storing your JSON as either VARCHAR (if you don't need special characters), NVARCHAR (if you need special characters, but this limits the string to 4000 characters unless you do NVARCHAR(MAX), which can store strings up to 2GB) or a BLOB. You can simply save/retrieve the JSON and parse it in your app, which is a much better approach since you can change the data being stored without having to update your database in the slightest (as in, not having to worry about adding/removing fields or changing their type).
The problem with that is the querying. Most SQL databases don't have native support for querying json and you resort to string comparison, which is relatively slow and not to mention error-prone.
Actual document databases allow very simple querying of json data (see for example http://docs.mongodb.org/manual/core/crud-introduction/).
|
On May 23 2015 23:33 supereddie wrote:Show nested quote +On May 23 2015 21:53 Manit0u wrote:On May 23 2015 19:27 supereddie wrote:
I'd just recommend using a differente type of database. Like a Document database. That way jou can store the JSON as... json. Easy. No mapping required. Yeah. You can do that in SQL by simply storing your JSON as either VARCHAR (if you don't need special characters), NVARCHAR (if you need special characters, but this limits the string to 4000 characters unless you do NVARCHAR(MAX), which can store strings up to 2GB) or a BLOB. You can simply save/retrieve the JSON and parse it in your app, which is a much better approach since you can change the data being stored without having to update your database in the slightest (as in, not having to worry about adding/removing fields or changing their type). The problem with that is the querying. Most SQL databases don't have native support for querying json and you resort to string comparison, which is relatively slow and not to mention error-prone. Actual document databases allow very simple querying of json data (see for example http://docs.mongodb.org/manual/core/crud-introduction/).
You're obviously right. I've only mentioned that in the context of using JSON with an SQL database. Anyway, the best solution when using SQL db would be to create some custom JSON parser that you could customise to your liking. Grab the JSON, parse it (preferably returning an object of the type you have in your entity model for the db) and then do the db operations.
Edit: Also, to refer to the previous posts, you should really avoid this:
for x in y
conn.execute(query);
What you should do is that:
query = "INSERT INTO (column1, column2, ...) VALUES";
subquery = ""; // put your subquery builder function here
// create a function here to build your "(value1, value2, ..), (value3, value4, ...), ..."
for x in y
subquery += "somevalue, "; // greatly simplified for the purposes of this presentation
conn.execute(query + subquery);
This way you only make a single query to update multiple rows in a table. Another thing to look into with this would be parametrizing the query.
|
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
Well, for me it's usually doing the same thing I do for work but I'm trying to do it in a different language, use different technologies etc. This way I'm not developing blindly as I already know what I want and how it should work, using different set of tools and methods for it lets me not only get familiar with other technologies, it also improves my ability to use the stuff I'm working with in my job as I get to see various approaches to the same thing and I can then use the best way to do it.
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
I'm developing the best PHP Framework that ever walked the tubes of the internet because I'm tired of the crappy PHP Frameworks that are available. I've tried most of them, basically all of the big ones, and all of them are just terrible, so I'm doing it right now with proper view models, hierarchical structure, full separation of views, controllers and models, not the pseudo-separation that all the frameworks seem to like, fully configurable, ...
Of course it's a long way away from perfection, especially considering that I'm currently in the third rewrite of almost the whole structure because I had a great idea that didn't fit into the existing structure.
Though... I think I spent more work time on it every time I was bored or finished a task faster than expected than actual time out of work.
|
On May 23 2015 17:10 Sufficiency wrote:Show nested quote +On May 23 2015 14:05 Blisse wrote:I don't know exactly what you're doing, but I don't see how you could ever reasonably write a perfectly generic JSON reader. And if it's not perfect then you'd have to parse the JSON yourself anyways, and parsing the JSON manually is EXTREMELY straightforward and easy. All you need to do is feed your parsed data into another function that inserts into the table. I REALLY don't see where the improvements will come from atm because I have no clue what you're parsing or inserting, but in my view from the code you've provided, I don't think you're doing things optimally yourself right now anyways?
def insertBooks(author_id, name, title, isbn):
statement = "INSERT INTO BOOKS (author_id, book_title, book_isbn) values ({0}, '{1}', '{2}');".format(int(author_id), title, isbn)
sql.execute(statement)
def parse():
for book in json['authors']['books']:
insertBooks(json['authors']['author_id'], book['title'], book['isbn'])
if all parse steps look like that it should be really fast to do, but again I don't know what you're dealing with. But again, writing that above took like 2 minutes. For just 3 columns, I agree. But note that your code has 2 fields with quotes and one without quote. Also your code will throw a SQL error if the JSON strings have single quote. Also it doesn't do well when one of the fields is missing. If a field is missing, which ones can we ignore and which ones are severe enough that we need to stop everything? Now imagine there are more than 50 columns, all of which may or may not have their own perks. Do I need to construct all of these cases by hand? This is why I want a more systematic approach. It makes my work faster and saves time because I will require less time testing. It does seem like an over-design; I don't disagree with you at all. But I kind of see the perspect of this.
I just took your pseudo code. You can just Url Encode all strings you receive. Your DB should have settings that allow null inserts.
If there are 50 columns then again it'll take like 5-15 minutes to turn the JSON into inputs to the function given you know what you want to store (given a bunch of functions you have to write). For the most part, each JSON schema is structured in a straightforward and human-readable manner (given how they're generated and why they're popular). The difference is that multiple schemas won't have exactly similar structures, so your systematic approach still requires you to figure out the logic of each new JSON schema and attach it to your all-in-one reader which will probably just end up being a giant if-tree that looks for key matches, that you still have to contribute to over time because you'll encounter new key names and values that need to be mapped or iterate through the content differently.
In the end all you're really doing is making a Book object and passing it to your DBMS to insert and deal with the failures in the Book object. I don't see making the a bunch of Book objects from JSON as a very difficult task, especially in Python. Doing it 100 times would be a chore, but chances are the structures are similar enough that the code re-use is possible sometimes. The alternative being what you described would be an extremely complicated procedure that would almost definitely take much more time than decomposing each schema yourself (given the Book object abstraction). That's why I say doing it would be more academic than practical (hence a bit idea). It might make your future work faster (or maybe not, because you still need to update it for new schemas).
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
Well I guess the problem is, that everything a single person can write in his/her free time is already done and can be downloaded for free.  I believe It's gonna be really hard to find something that's really useful AND doable in a reasonable amount of time.
I used to program small games or even a small game 2d game engine (based on another engine, though :D). I guess its nothing really useful but it was fun and its a nice thing you can share with friends. But lately I didn't do any free time programming projects. I was doing other stuff, that fun too like music and sport. I guess 40 hours of programming a week is enough for me.
P.S.: Programming exercises are not boring if you learn something new and interesting  Have you tried some meta programming or logical programming? You could also try to define you own little language for a certain type of program. Or if you like programming AIs here is a fun and competitive project http://codecombat.com/
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
Right now I'm writing a patch to an Emacs extension for on-the-fly syntax checking, to let it autodetect compiler flags from a file :D so I can shut up all the VS2015 fanbois at work lol. Generally, I write games in my spare time, I also did a really shitty operating system a while back which was really interesting (way too much work to do a real OS on your own though lol). I write useful things enough at work, I like to do fun things on my own time 
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
None.
First, I'm usually pretty worn out after a day of work, not interested in doing more software.
Second, technically everything I work on is owned by the company, unless I file the correct paperwork beforehand. It wouldn't be that bad to file, and actually even if I didn't file nobody would really care as long as I don't publicly release anything. But, just an extra step of annoyance.
If / when both of those were not true, I wouldn't even really be doing much programming - I'd be going back to doing embedded systems (I was originally EE, not software). Like I'd want to build my own RF-transmitter sensor networks for around the house, detecting open/unlocked windows & doors, etc.
|
On May 24 2015 11:53 phar wrote:Show nested quote +On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate. None. First, I'm usually pretty worn out after a day of work, not interested in doing more software. Second, technically everything I work on is owned by the company, unless I file the correct paperwork beforehand. It wouldn't be that bad to file, and actually even if I didn't file nobody would really care as long as I don't publicly release anything. But, just an extra step of annoyance. If / when both of those were not true, I wouldn't even really be doing much programming - I'd be going back to doing embedded systems (I was originally EE, not software). Like I'd want to build my own RF-transmitter sensor networks for around the house, detecting open/unlocked windows & doors, etc.
I usually put embedded systems in the realm of software though, just more towards the "cool" side of things :p
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
I have a list of things I want to do when I have the free time that I keep updating.
Mainly I focus on coding things close at hand but I juggle a couple of side projects. Sometimes apps need updating, cleaning up, and such. Maybe I'll write a library. Maybe something on HackerNews sounded really cool and I'll take a look at the repo to see if I can contribute. Mobile apps are the easiest once you start "knowing" how to do things. Same with desktop apps. Lately I want to build bigger more connected systems (phone, tablet to desktop) but I haven't had the time with schoolwork.
But it's literally just do it. When you have free time you spend it on hobbies (or waste it browsing the web or posting to TL all the time ). Coding projects are just forcing yourself to make programming a hobby as well as a job. If it's vastly different fromy our normal university/schoolwork, it's a lot easier to get into in my opinion.
I also choose cool things, not things that are necessarily useful, but things I think would be neat. Have a low bar for neat means you can always have ideas.
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
I wrote simple arbitrary precision bignum arithmetic libraries for R (using Rcpp) and C++, mostly because everything I could find online either didn't work or was needlessly complicated to implement (seriously wtf, gmp), or wasn't truly arbitrarily precise (just allowing for arbitrarily long inputs).
I just like to play with big numbers and wanted a simple framework for that.
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
I'm slowly working on rewriting my blog, instead of using Wordpress. Using Express 4.0 and MongoDB on the backend (which I've got some experience with from a previous job), and deciding if I should learn AngularJS for the frontend, with maybe some Bootstrap CSS. Leaning towards doing so, as I figure it's a good idea to learn some new stuff.
Once that is done I plan on making small mobile games and just seeing how that goes.
I use a lot of Java/AS3/ some PHP at work, so using different languages at home for my personal stuff can make for a nice change. That said, I haven't really done too much at home since finishing my degree and starting full time work, partner/video games/dogs are highly distracting.
|
On May 24 2015 01:56 darkness wrote:
What projects do you guys do outside university and work? Every time I have some time to code something, I don't have an idea for a personal project. I don't do programming exercises as they're boring, but I can't come up with something practical that I need to use/automate.
Did write some programs for my Raspberry Pi (mainly automation of standard stuff I use often).
In the future I want to get my hands on some stuff like those RGB light bulbs (Philips Hue, Mi Light, ... (haven't decided)) and do some automation of that sort (change color of lights when certain events occur). Someone else mentioned sensors, those might have a place in it as well.
I wrote a simulation my parents use for their hobby company. If I had any PHP skills I would probably help with their website overall.
I have old school friends (architect, doctor) and I rarely do some stupid / fun stuff with them in visualization / simulation (think of rough prototypes/poc, nothing real).
I like to play boardgames, so if I had more time I might consider writing some apps for those, making play easier.
|
Once again I struggle with c++, this time its std::maps.
I have multiple maps which go like:
std::map < std::pair<int key1, int key2>, int value> myMap
I struggle with writing a function to retrieve key1 and key2 from the map, so that another class can use them. Each map is hold as an object.
What I need to do is to merge mutliple maps while adding values with the same keypair, so map::insert wont cut it. For example:
myMap1[std::make_pair(11, 12)] = 20;
myMap1[std::make_pair(30, 32)] = 30;
myMap2[std::make_pair(11, 12)] = 20
myMap3 = myMap1 + myMap2 should be:
11,12 = 40
30, 32 = 30
I am searching for a way to iterate through a map, step by step since atm, I am only able to iterate thorugh it as a whole with a for-loop. I believe I need iterators for map1 and 2. Both would start at the first element of each map, I would then run iterator2 through map2 until it either finds a keypair that matches the one at iterator1's position in map1, or one thats bigger(since map is sorted by default, searching past this point for a matching keypair to iterator1's position would be useless). If a matching keypair is found, I just add the value, otherwise I use map::insert. iterator1 will then go to the next element of map1 and interator2 can continue from his last position.
In theory/with having direct access to both maps and having to deal with classes, I am able to do all of this, I just need some help with my iterator in combination with having the maps in different objects. A solution that i thought of is just having a vector in each object containing a map, that holds all valid keypairs, since I would be able to work with that, but this seems like a rubbish solution.
|
Could you clarify? It's not really clear what you mean by "merge mutliple maps while adding values with the same keypair".
If you want to add merge, just take map A and insert or add every key in map B depending on whether the key in map B exists in map A, and return map A. The type of key doesn't matter.
|
My problem is, when using map::insert with 2 maps, it will only insert the key-value-pairs of map2 into map1, that arent already present. For example:
myMapA[0] = 10
myMapA[1] = 20
myMapA[2] = 30
myMapB[0] = 2
myMapB[1] = 4
myMapB[7] = 6
myMapA.insert(myMapB.begin(), myMapB.end());
for (auto elem : myMap3)
{
std::cout << elem.first << " " << elem.second << "\n";
}
will give you:
myMapA[0] = 10
myMapA[1] = 20
myMapA[2] = 30
myMapA[7] = 6
Whenever a key already exists in one map (in this case [0] and [1], I would have to do;
myMapA[0] = myMapA[0] + myMapB[0]
My problem is, I need the key which is in both maps to add this, but I dont know how, without going thorugh the whole map first, writing the keys into an container and use a get-Method to hand them to the class that manages my maps.
|
|
|
Guys we should do some project together just for fun. Preferably (for me) C++ as I find it fun, challenging and I have so much to learn with that language. I don't mind C# or other OOP language if the majority don't want C++. What do you say? We can put it to a vote if you want.
|
What are you planning exactly? The language is irrelevant until you have an idea what is being built.
|
On May 27 2015 06:40 Nesserev wrote:
...
for (auto& pair : myMapB)
myMapA[pair.first] = myMapA[pair.first] + myMapB[pair.first];
...
You can probably shorten that a bit:
for (auto const& pair : myMapB)
myMapA[pair.first] += pair.second;
(You're not changing "pair", so use const& instead of &. No need to do the myMapB lookup, since the value is in pair. No need to do two lookups in myMapA (though I suspect that might get optimized anyway)).
I'd be up for collaborating on something, if the project is interesting.
|
|
|
|
|
|
EPT
