EPTThe Big Programming Thread - Page 877
| Forum Index > General Forum |
Thread Rules 1. This is not a "do my homework for me" thread. If you have specific questions, ask, but don't post an assignment or homework problem and expect an exact solution. 2. No recruiting for your cockamamie projects (you won't replace facebook with 3 dudes you found on the internet and $20) 3. If you can't articulate why a language is bad, don't start slinging shit about it. Just remember that nothing is worse than making CSS IE6 compatible. 4. Use [code] tags to format code blocks. | ||
|
Manit0u
Poland17750 Posts
| ||
|
Acrofales
Spain18296 Posts
On April 26 2017 12:55 Manit0u wrote: This definitely gets me closer. I'll still need to figure out some more stuff (obviously, the system is a bit more complex than that, with different queues, pipelines, engines, multiple statuses like rejected, blocked etc. and I need to scope it out properly). If that's the case is it really worth doin in SQL? | ||
|
solidbebe
Netherlands4921 Posts
The idea for now is to use javascript and the D3 library to make the visualization. D3 seems very flexible and running it in the browser means we don't have to worry about different versions for OS and updating is easy. I have pretty much no experience with this kind of project so any suggestions are welcome. | ||
|
Manit0u
Poland17750 Posts
On April 26 2017 21:59 solidbebe wrote: So I'm doing a project where we are making a system that documents and visualizes system architecture and information flow at a company (which uses 3000+ different IT systems). I'm looking for inspiration but I'm finding it very difficult to find examples of projects similar to this online. I just get flooded in UML/sequence diagrams which is obviously not helpful. Does anyone know of anything? The idea for now is to use javascript and the D3 library to make the visualization. D3 seems very flexible and running it in the browser means we don't have to worry about different versions for OS and updating is easy. I have pretty much no experience with this kind of project so any suggestions are welcome. I'd suggest using neo4j database for that (awesome for graphs) and then you could check facebook graph api for how they generate their output: ![[image loading]](http://code-worrier.com/images/graph_visualization.jpg) | ||
|
Djagulingu
Germany3605 Posts
On April 26 2017 21:59 solidbebe wrote: So I'm doing a project where we are making a system that documents and visualizes system architecture and information flow at a company (which uses 3000+ different IT systems). I'm looking for inspiration but I'm finding it very difficult to find examples of projects similar to this online. I just get flooded in UML/sequence diagrams which is obviously not helpful. Does anyone know of anything? The idea for now is to use javascript and the D3 library to make the visualization. D3 seems very flexible and running it in the browser means we don't have to worry about different versions for OS and updating is easy. I have pretty much no experience with this kind of project so any suggestions are welcome. 3000+? Is that real? That number is 50 times the amount of different IT systems one of the telecommunications giants in my country uses. About the project: If you are going to draw anything on the web, you can't ever go wrong with D3. We have done something similar by having JSON objects as blueprints and drawing with D3 according to the blueprint. In such a structure, SVG is much more efficient than Canvas, so D3 is the way to go. Also, don't ever think about server side rendering with this shit. | ||
|
spinesheath
Germany8679 Posts
On April 26 2017 21:59 solidbebe wrote: So I'm doing a project where we are making a system that documents and visualizes system architecture and information flow at a company (which uses 3000+ different IT systems). I'm looking for inspiration but I'm finding it very difficult to find examples of projects similar to this online. I just get flooded in UML/sequence diagrams which is obviously not helpful. Does anyone know of anything? The idea for now is to use javascript and the D3 library to make the visualization. D3 seems very flexible and running it in the browser means we don't have to worry about different versions for OS and updating is easy. I have pretty much no experience with this kind of project so any suggestions are welcome. Maybe gephi.org is of any help? | ||
|
Deleted User 3420
24492 Posts
A class has 5 TAs and 2 teachers. A team of 4 of them is assigned (at random) to grade papers. What are the odds that the 2 teachers are assigned to grade papers? Thee question came with a hint that for this one they wanted a ratio of natural numbers (fraction), and that the answer should be found by using fractions and using a process of canceling most of them out. The hint confused the shit out of me. The only way I see to solve the problem, I THINK, is to do all the combinations where you hit the 2 teachers in 4 tries which reasonably seems like I should use the number of permutations with the 2 teachers in 4 letters out of the total number of permutations. but calculating this uses factorials which they said not to do I don't really know how to do it otherwise. something like multiplying out every actual chance of hitting the teacher each try for every combination of 4 tries, which seems fucking ridiculous so what am I not getting about that, it makes me mad. we never did anything like that in class | ||
|
WolfintheSheep
Canada14127 Posts
Total combinations with 2 Teachers, divided by total combinations. So like (5C2 + 2C2) / (7C4) or something. | ||
|
Deleted User 3420
24492 Posts
and I think that 5c2 makes sense for a numerator (you wouldn't add 2c2, you would multiply it.. I think) but the stupid hint told us to use natural number fractions(meaning don't use exponents or factorials or no choose r... it literally said that). so i shouldn't be working in this way at all. and more specifically it told us to cancel fractions out. I think he really fucked me over with his "hint" is what happened when I did the test I did solve towards 5c2/7c4 at first until i re-read it and saw his "hint" | ||
|
WolfintheSheep
Canada14127 Posts
((5*4)/2) / ((7*6*5*4)/(4*3*2)) From there you can separate things out and cancel. | ||
|
Acrofales
Spain18296 Posts
On April 27 2017 03:27 travis wrote: denominator would be 7c4, yes and I think that 5c2 makes sense for a numerator (you wouldn't add 2c2, you would multiply it.. I think) but the stupid hint told us to use natural number fractions(meaning don't use exponents or factorials or no choose r... it literally said that). so i shouldn't be working in this way at all. and more specifically it told us to cancel fractions out. I think he really fucked me over with his "hint" is what happened when I did the test I did solve towards 5c2/7c4 at first until i re-read it and saw his "hint" So write out your factorials and do the math, lol. Lots will cancel out and you're left with a normal fraction. You know, write out 5c2 as 5! / 2!*3!, etc. | ||
|
Deleted User 3420
24492 Posts
but yeah I see where you are going I could have gotten all the way here just fine (well I would have left it in the form of n-choose r's). But the hint pretty much told us not to use n-choose r at all so yeah I get 0 points even though I could have solved this fine and I am going to be pretty mad On April 27 2017 03:46 Acrofales wrote: So write out your factorials and do the math, lol. Lots will cancel out and you're left with a normal fraction. You know, write out 5c2 as 5! / 2!*3!, etc. I'll quote the "hint" for you when I get my exam back and you will understand my complaint | ||
|
Acrofales
Spain18296 Posts
Then math the whole thing out. | ||
|
Deleted User 3420
24492 Posts
I mean.. it's my failure though either way in that I didn't just have more confidence in that I still was getting the correct result regardless. But it sucks to lose points when you know material. | ||
|
Zocat
Germany2229 Posts
On April 27 2017 03:27 travis wrote: denominator would be 7c4, yes and I think that 5c2 makes sense for a numerator (you wouldn't add 2c2, you would multiply it.. I think) You are correct, it would be *2c2. You have a Hypergeometric distribution. You have learned the urn model, right? If not, do that it will make all your combinatorics way easier. You break down your problem and find the correct urn model. Then look up what kind of distribution is associated with that model. | ||
|
Manit0u
Poland17750 Posts
We have: A has many B, C has many B. Now, we do filtering on C matching A where C has no B or CB are a subset of AB. The thing is, it's pretty slow as soon as you hit about a million records in the db (1.5s query), which is no good for us. Any ideas how can you optimize it in postgres? Right now we have join tables that are being aggregated into views (stale data is unacceptable since it's used for live time pooling and assigning C to A with race conditions and all that jazz). Edit: I'm seriously considering dropping the join tables and simply dumping all the related ids into an uuid[] column in respective tables. | ||
|
enigmaticcam
United States280 Posts
On April 27 2017 16:31 Manit0u wrote: Back to SQL again... With the problem I posted some time back. We have: A has many B, C has many B. Now, we do filtering on C matching A where C has no B or CB are a subset of AB. The thing is, it's pretty slow as soon as you hit about a million records in the db (1.5s query), which is no good for us. Any ideas how can you optimize it in postgres? Right now we have join tables that are being aggregated into views (stale data is unacceptable since it's used for live time pooling and assigning C to A with race conditions and all that jazz). Edit: I'm seriously considering dropping the join tables and simply dumping all the related ids into an uuid[] column in respective tables. If I remember correctly, SQL is a bit slow on OR statements. Maybe do the "Has no B" and the "CB are a subset of AB" in separate queries and union them. Also, in my experience I've found that when I have to do complex joins or filters between large datasets, sometimes it helps to do as much filtering as you can first and insert into temp tables with indexes, and then do joins from there. Not sure if that will help you here. | ||
|
enigmaticcam
United States280 Posts
Why does this script only take 1 second to run...
...and yet this script takes about 10 minutes.
The table size is about 22 million rows. So I know the issue is that one is doing a distinct on the entire table, and one is doing a distinct only on the temp table. But I would think the second one would first filter for the smaller subset before performing the distinct, but clearly it's not. Is there a reason for that? | ||
|
Thaniri
1264 Posts
If my understanding is right, you can have a worst case of 22m * 22m results. The first script doesn't have a subquery at all, so as a whole it never has to be re-executed. Mind you, I'm not a professional. This is me just regurgitating what I learned from one class where we happened to learn about these optimizations. edit: You could make it even faster without using a temp table, but perhaps for safety or concurrency reasons you might want to still use it. | ||
|
Manit0u
Poland17750 Posts
On April 28 2017 01:59 enigmaticcam wrote: If I remember correctly, SQL is a bit slow on OR statements. Maybe do the "Has no B" and the "CB are a subset of AB" in separate queries and union them. Also, in my experience I've found that when I have to do complex joins or filters between large datasets, sometimes it helps to do as much filtering as you can first and insert into temp tables with indexes, and then do joins from there. Not sure if that will help you here. We've removed the join columns entirely now. Switched to just storing ids in an array as a table column. It's super effective but required us to implement some back-end mechanisms for syncing stuff (not that hard since we're only interested in a scenario when a record whose id should go into the array is deleted which should be a rare case). | ||
| ||
