|
Thread Rules
1. This is not a "do my homework for me" thread. If you have specific questions, ask, but don't post an assignment or homework problem and expect an exact solution.
2. No recruiting for your cockamamie projects (you won't replace facebook with 3 dudes you found on the internet and $20)
3. If you can't articulate why a language is bad, don't start slinging shit about it. Just remember that nothing is worse than making CSS IE6 compatible.
4. Use [code] tags to format code blocks. |
On August 21 2019 04:12 WarSame wrote:
Technically agile can deal with that, but that does sound very unhealthy. What could you be doing in such a large project that you will be asked to switch multiple times in a day?
So I can't answer for him, obviously, but I can answer with an anecdote from my own previous job. Our boss wasn't much of a programmer, so he would mainly deal with the customers. The problem was that every time he talked to one of the many customers, he would get an input, and immediately tell one of the programmers to fix that. No backlog, no queue, just an immediate focus shift.
Now agile is of course made to deal with exactly that. However we didn't use agile, we didn't even use waterfall. We were just told what to do from our boss and started working on that.
That, along with a multitude of other issues (For instance: The program would literally mean life and death for its users, and we weren't allowed to write tests because they took focus away from new features) forced me to find a new job. It was by far the worst workplace I have ever been at.
|
On August 21 2019 00:41 Excludos wrote:Show nested quote +On August 20 2019 22:54 Neneu wrote:On August 20 2019 21:31 Silvanel wrote:The project is development of new generation of SW for Daimler (mostly Mercedes, but not limited to it) cars. So Excludos You work for Intel right? (judging by the revenue)As to the agile fail it is complicated issue obviously. I do not feel comfortable discussing details on internet. I will just say that Daimler is both one of the most damanding and better paying customers. Combine that with enourmous size od the project, dependenace on many external SW and HW suppliers (including Intel btw  ), set in stone deadlines, extremely short feedback loops it turned out there is no agile framework suitebale for project of this size. Still we use some "agile" parts, just not the whole frameworks. Think it is more likely he works for Equinor (if I remember correctly they had about 61ish $ billion revenue in 2017). Now I don't know how it is working with them, but from my own experience in Stavanger, almost every single IT company or company with its own IT department uses agile frameworks here. Yeah. I think I even said so earlier. I'm a consultant, but my current project is with Equinor. Like you said, I don't think there's a single company left in Norway that doesn't use it. It's just superior in every way really. But I know a lot of other countries are slow to adapt. A lot because they don't have the know-how, and a lot because they're driven by older business men who doesn't understand it, and doesn't like change.
I think you're way over generalizing. I modernize/overhaul/rebuild Foxpro applications built from 1986 to 2010 in Canada and the north eastern USA. These projects were almost all built under agile project management. I'd say half of them were coded inside what would today be described as an agile application framework. The next biggest % of them used no framework at all. Any how, the management at the time probably just didn't call it "agile".... but that's what was happening. They probably had some other 7 syllable word for their management style that they used to try and justify their $200/hour rates.
|
deep learning question:
it is stated by the universal approximation theorem: "that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of Rn, under mild assumptions on the activation function." (taken from wikipedia)
the general claim is that activation functions are required to approximate a solution to many problems because if we add a bunch of lines together we are just going to get another line as an output, and obviously we can't approximate many functions with a line.
but my question is if there is an alternative method where linear activations *can* approximate any continuous function on compact subsets of Rn. this method would be through *feature construction*.
for example, say we have 1000 training samples of 28x28 images we want to classify
now if there is any level of complexity to this you won't typically be able to perform well with linear activations
however, what if we "solve" all sub-features of our training samples. what if instead of using our 28x28 inputs, we also construct every single (28x28)! combination of inputs and use each constructed combination as another input.
we then perform linear regression on these samples, each containing (28x28)! features.
is it not the case that this would provide optimal accuracy?
thanks
|
Deeplearning is about real world applications. Ican't figure out the connection to (28x28)!
|
I've scoured the internet for a simple code but I cannot find it.
Does anyone here know how I can read solar panel consumption and discharge using Arduino?
I have a MCP73871 chip connected to a breadboard, connected to arduino. Panel and battery all connected as well.
I just want to get a simple reading on what the solar panel is doing and if it's enough to slow charge a hefty battery.
|
What data exactly are you interested in? How much volt the solar panel outputs? How much power? Do you want to observe this while doing something, or just measure it once?
|
Yes to literally all of that. I want to see initial readings to know it's working but I also want to check a "full day."
|
Firstly, i don't know anything about that chip. However, i do know a bit about the Arduino.
For initial readings, you could just attach it to the Analog ports of the arduino, set that port to input, and read the voltage value of that port. I doubt that the cell delivers more than 5V. For voltage over a day, you can do the same, but need to log it in the programming, or sum it up.
Power might be a bit harder. You could simply attach a low ohm resistor, and measure the voltage that is on that resistor due to the solar cell. If you know the value of the resistor, you can calculate the current, and thus the power that ends up being used in the resistor, which should be all of the power the cell provides. Use a low ohm resistor to make sure that you actually use all the power which the cell can deliver.
If you want to do stuff with the chip, you probably need to find some library that knows how to talk to it.
|
The chip is just to use the solar panel to charge a battery. It doesn't do much more than that (from what I'm reading.)
The solar cell is 5V with a .7w output. The battery is a 12V 3000mAh Ni-MH battery. I have a lot of other stuff hooked up to it that requires a hefty battery to charge it all. The output of the solar cell is very small, but if it trickles into the battery and keeps it charged over a period of time, then it will solve my current issues (I'll need to develop a more efficient way to charge and store power, but I'm prototyping at the moment.)
I'll try to attach it to the A4 port of the Arduino and see what I can come up with.
Is there any code that I need to read it? I never went too far with Arduino so I'm just being extra cautious to not burn anything out that would put me back. Thanks Sim.
|
I assume you have something like this with two wire DC output?
https://www.amazon.com/Small-Solar-Panel-4V-100mA/dp/B002MAUYRK
Reading over the datasheet for the MCP73871:
http://ww1.microchip.com/downloads/en/DeviceDoc/MCP73871-Data-Sheet-20002090E.pdf
On page 6 "Input Current Limit Control (ICLC)" is probably what you want to use.
You'll probably want a voltage regulator in front of this from the solar panel, powering V_in at 5V since the solar panel's voltage isn't constant.
If you read the solar panel's supply voltage, before and after connecting the MCP, and the voltage drops then there's current running through the MCP/charging the battery. To find out how much, you'd need to know the current which would be difficult to find without an ammeter.
A .7W output should be enough to charge it at the supply current limit assuming some loss (see page 6 of the datasheet)
Setting it to
PROG2 = High, SEL = Low
TA = -5°C to +55°C
You get a 5v, 450mA output which is ~2.5w charge rate.
The bigger problem is the 12V+ output you'd need to charge a 12V battery
The MCP73871 device employs a constant
current/constant voltage (CC/CV) charge algorithm
with selectable charge termination point. To
accommodate new and emerging battery charging
requirements, the constant voltage regulation is fixed
with four available options: 4.10V, 4.20V, 4.35V or
4.40V
You might blow this up trying to charge it since the battery voltage is so much higher than the controllers.
Edit::
https://www.digikey.ca/products/en/integrated-circuits-ics/pmic-battery-chargers/781?k=battery charger
This page separates them by chemistry as well as cells. You'll need:
1. A second solar panel to wire in series for 14V output
2. A PMIC that can output probably close to 14V.
|
On August 31 2019 02:05 mahrgell wrote:
Deeplearning is about real world applications. Ican't figure out the connection to (28x28)!
im asking a theory question
(28x28) is the amount of inputs in the example. so, 784 inputs
and oops about 784!, I think I actually mean 2^784. What I am trying to say is the count of all possible subsets of our set of 784 inputs.
|
On August 31 2019 07:36 Lmui wrote:I assume you have something like this with two wire DC output? https://www.amazon.com/Small-Solar-Panel-4V-100mA/dp/B002MAUYRKReading over the datasheet for the MCP73871: http://ww1.microchip.com/downloads/en/DeviceDoc/MCP73871-Data-Sheet-20002090E.pdfOn page 6 "Input Current Limit Control (ICLC)" is probably what you want to use. You'll probably want a voltage regulator in front of this from the solar panel, powering V_in at 5V since the solar panel's voltage isn't constant. If you read the solar panel's supply voltage, before and after connecting the MCP, and the voltage drops then there's current running through the MCP/charging the battery. To find out how much, you'd need to know the current which would be difficult to find without an ammeter. A .7W output should be enough to charge it at the supply current limit assuming some loss (see page 6 of the datasheet) Setting it to PROG2 = High, SEL = Low TA = -5°C to +55°C You get a 5v, 450mA output which is ~2.5w charge rate. The bigger problem is the 12V+ output you'd need to charge a 12V battery The MCP73871 device employs a constant current/constant voltage (CC/CV) charge algorithm with selectable charge termination point. To accommodate new and emerging battery charging requirements, the constant voltage regulation is fixed with four available options: 4.10V, 4.20V, 4.35V or 4.40V You might blow this up trying to charge it since the battery voltage is so much higher than the controllers. Edit:: https://www.digikey.ca/products/en/integrated-circuits-ics/pmic-battery-chargers/781?k=battery chargerThis page separates them by chemistry as well as cells. You'll need: 1. A second solar panel to wire in series for 14V output 2. A PMIC that can output probably close to 14V.
I have this solar cell: https://www.amazon.com/gp/product/B07JLX1KDJ/ref=ppx_yo_dt_b_asin_title_o09_s00?ie=UTF8&psc=1
and this battery: http://www.revrobotics.com/rev-31-1302/
I've attached a Capacitor to the MCP and I think I have some MOSFETs somewhere that should help with the regulation. Since the application demands that everything be as miniature as possible, this is going to be some very customized work after I get it all working. I only bought the small solar panel to test and would indeed plan on getting something a bit more robust to handle the load, but I'm still searching for something that will suit my requirements. The goal is to keep the total cost of construction low as it would tack a premium on the end result.
I'll read the links you provided and see if my electrical engineer friend can draw a schematic for me to follow, but it seems to be getting a bit too complicated for me to progress on my own.
|
On August 31 2019 10:52 ZerOCoolSC2 wrote:Show nested quote +On August 31 2019 07:36 Lmui wrote:I assume you have something like this with two wire DC output? https://www.amazon.com/Small-Solar-Panel-4V-100mA/dp/B002MAUYRKReading over the datasheet for the MCP73871: http://ww1.microchip.com/downloads/en/DeviceDoc/MCP73871-Data-Sheet-20002090E.pdfOn page 6 "Input Current Limit Control (ICLC)" is probably what you want to use. You'll probably want a voltage regulator in front of this from the solar panel, powering V_in at 5V since the solar panel's voltage isn't constant. If you read the solar panel's supply voltage, before and after connecting the MCP, and the voltage drops then there's current running through the MCP/charging the battery. To find out how much, you'd need to know the current which would be difficult to find without an ammeter. A .7W output should be enough to charge it at the supply current limit assuming some loss (see page 6 of the datasheet) Setting it to PROG2 = High, SEL = Low TA = -5°C to +55°C You get a 5v, 450mA output which is ~2.5w charge rate. The bigger problem is the 12V+ output you'd need to charge a 12V battery The MCP73871 device employs a constant current/constant voltage (CC/CV) charge algorithm with selectable charge termination point. To accommodate new and emerging battery charging requirements, the constant voltage regulation is fixed with four available options: 4.10V, 4.20V, 4.35V or 4.40V You might blow this up trying to charge it since the battery voltage is so much higher than the controllers. Edit:: https://www.digikey.ca/products/en/integrated-circuits-ics/pmic-battery-chargers/781?k=battery chargerThis page separates them by chemistry as well as cells. You'll need: 1. A second solar panel to wire in series for 14V output 2. A PMIC that can output probably close to 14V. I have this solar cell: https://www.amazon.com/gp/product/B07JLX1KDJ/ref=ppx_yo_dt_b_asin_title_o09_s00?ie=UTF8&psc=1and this battery: http://www.revrobotics.com/rev-31-1302/I've attached a Capacitor to the MCP and I think I have some MOSFETs somewhere that should help with the regulation. Since the application demands that everything be as miniature as possible, this is going to be some very customized work after I get it all working. I only bought the small solar panel to test and would indeed plan on getting something a bit more robust to handle the load, but I'm still searching for something that will suit my requirements. The goal is to keep the total cost of construction low as it would tack a premium on the end result. I'll read the links you provided and see if my electrical engineer friend can draw a schematic for me to follow, but it seems to be getting a bit too complicated for me to progress on my own.
Good luck.
Assuming you have a PMIC, your next step to measuring exact power would be to get a ammeter/voltmeter and attach it to the battery leads so you can calculate the power going to the battery. You may be able to guesstimate the battery charge level without that using a voltmeter at idle conditions for the battery as well if knowing whether the battery is full.
Also:
https://batteryuniversity.com/learn/article/charging_nickel_metal_hydride
NiMH batteries don't do well with slow charging apparently because and low C (Amperage) rates make it hard to calculate the full condition because the voltage change isn't clearly defined.
Side note - I found this again:
http://www.reddit.com/r/programming/comments/9x15g/programming_thought_experiment_stuck_in_a_room/c0ewj2c
still a great read through, and a nightmare to think about.
|
On September 04 2019 06:07 Lmui wrote:Show nested quote +On August 31 2019 10:52 ZerOCoolSC2 wrote:On August 31 2019 07:36 Lmui wrote:I assume you have something like this with two wire DC output? https://www.amazon.com/Small-Solar-Panel-4V-100mA/dp/B002MAUYRKReading over the datasheet for the MCP73871: http://ww1.microchip.com/downloads/en/DeviceDoc/MCP73871-Data-Sheet-20002090E.pdfOn page 6 "Input Current Limit Control (ICLC)" is probably what you want to use. You'll probably want a voltage regulator in front of this from the solar panel, powering V_in at 5V since the solar panel's voltage isn't constant. If you read the solar panel's supply voltage, before and after connecting the MCP, and the voltage drops then there's current running through the MCP/charging the battery. To find out how much, you'd need to know the current which would be difficult to find without an ammeter. A .7W output should be enough to charge it at the supply current limit assuming some loss (see page 6 of the datasheet) Setting it to PROG2 = High, SEL = Low TA = -5°C to +55°C You get a 5v, 450mA output which is ~2.5w charge rate. The bigger problem is the 12V+ output you'd need to charge a 12V battery The MCP73871 device employs a constant current/constant voltage (CC/CV) charge algorithm with selectable charge termination point. To accommodate new and emerging battery charging requirements, the constant voltage regulation is fixed with four available options: 4.10V, 4.20V, 4.35V or 4.40V You might blow this up trying to charge it since the battery voltage is so much higher than the controllers. Edit:: https://www.digikey.ca/products/en/integrated-circuits-ics/pmic-battery-chargers/781?k=battery chargerThis page separates them by chemistry as well as cells. You'll need: 1. A second solar panel to wire in series for 14V output 2. A PMIC that can output probably close to 14V. I have this solar cell: https://www.amazon.com/gp/product/B07JLX1KDJ/ref=ppx_yo_dt_b_asin_title_o09_s00?ie=UTF8&psc=1and this battery: http://www.revrobotics.com/rev-31-1302/I've attached a Capacitor to the MCP and I think I have some MOSFETs somewhere that should help with the regulation. Since the application demands that everything be as miniature as possible, this is going to be some very customized work after I get it all working. I only bought the small solar panel to test and would indeed plan on getting something a bit more robust to handle the load, but I'm still searching for something that will suit my requirements. The goal is to keep the total cost of construction low as it would tack a premium on the end result. I'll read the links you provided and see if my electrical engineer friend can draw a schematic for me to follow, but it seems to be getting a bit too complicated for me to progress on my own. Good luck. Assuming you have a PMIC, your next step to measuring exact power would be to get a ammeter/voltmeter and attach it to the battery leads so you can calculate the power going to the battery. You may be able to guesstimate the battery charge level without that using a voltmeter at idle conditions for the battery as well if knowing whether the battery is full. Also: https://batteryuniversity.com/learn/article/charging_nickel_metal_hydrideNiMH batteries don't do well with slow charging apparently because and low C (Amperage) rates make it hard to calculate the full condition because the voltage change isn't clearly defined. Side note - I found this again: http://www.reddit.com/r/programming/comments/9x15g/programming_thought_experiment_stuck_in_a_room/c0ewj2cstill a great read through, and a nightmare to think about.
Thank you! I don't see a PMIC on the MCP73871. In regards to the battery, is was the best option/price at the time, so I got that just for test purposes. I found other batteries that are smaller and would allow me to do what I am intending to, as well as keep a good charge. Like I mentioned, this whole setup is just for testing and seeing if I can power everything and if the components do what I need them to do. Admittedly, not everything will be included on the final design, but I'm attempting to future proof this with expandable options, should the need arise.
And to the programming experiment...Nah. I'm good ^_^/ I remember handwriting C++ and HTML back in the day before we got a computer at my house. I'd then to go my friend's dad and ask him to use his computer to code it in and see if it worked. Got a bit more versed in C++ and then stopped. Got versed in CSS and then stopped. Never got back into it. Now I'm looking at Java and I'm just...tired.
|
Request for Help!
How to scrape HTML data-endpoints with Beautiful Soup (modal containers)
I'm currently trying to scrape data from baseball reference and all is going well except I've hit a problem with trying to scrape data within a modal container/data-endpoint. Now, I actually don't know HTML, though I'm familiar enough to get by with scraping easily enough - which is why I'm not sure if I should be classifying what I'm trying to enter as a data-endpoint or a modal-container. Hopefully it will be made clear below:
If you go to https://www.baseball-reference.com/players/gl.fcgi?id=torregl01&t=b&year=2019 , an example page I'm trying to scrape from, and view the column labeled "PA", you'll see clicking on the value loads a pop-up window. I'm trying to scrape info from that window, which is what is causing issues for me - I simply don't know how to scrape information that isn't directly on the webpage.
My initial scraping of the page began with:
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")
Below is the html I've found for the container through "Inspect":
![[image loading]](https://i.stack.imgur.com/XPK9I.png)
I also found that the "PA" cell in each row has a data-endpoint listed in its properties shown below:
![[image loading]](https://i.stack.imgur.com/BYzbw.png)
I'm unsure how to access the information in the pop-up window and would prefer to do it using Beautiful Soup if possible. If someone could at least point me in the right direction as to scraping info off these types of pop-up windows that would be super helpful.
A huge thank you to anyone kind enough to help me - it will be very much appreciated!
EDIT: I've seen some sources that suggest I need to use Selenium, but I don't know how to use it lol
EDIT 2: Actually, that data-endpoint appended to the base url gives a new webpage with the desired information. I'll try to load that webpage as a separate "soup" - if that's possible.
|
You can also try just re-parsing the original page after opening the modal (if that is an option). Most modals work by appending html to existing page and displaying it.
|
On September 16 2019 20:28 Manit0u wrote:
You can also try just re-parsing the original page after opening the modal (if that is an option). Most modals work by appending html to existing page and displaying it.
Would be nice, but it doesn't seem like the original page is being appended to after displaying the pop-up. If I "Inspect" the pop-up window there is HTML code listed, but its parent node is the original page, so it only appears after clicking on the link for it. But appending the data-endpoint, which is available on the original page, to the base url brings me to the correct page with the info I need. So I plan to parse that new page in every iteration of the loop.
Anyway, thank you!
|
On August 31 2019 00:02 travis wrote:
deep learning question:
it is stated by the universal approximation theorem: "that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of Rn, under mild assumptions on the activation function." (taken from wikipedia)
the general claim is that activation functions are required to approximate a solution to many problems because if we add a bunch of lines together we are just going to get another line as an output, and obviously we can't approximate many functions with a line.
but my question is if there is an alternative method where linear activations *can* approximate any continuous function on compact subsets of Rn. this method would be through *feature construction*.
for example, say we have 1000 training samples of 28x28 images we want to classify
now if there is any level of complexity to this you won't typically be able to perform well with linear activations
however, what if we "solve" all sub-features of our training samples. what if instead of using our 28x28 inputs, we also construct every single (28x28)! combination of inputs and use each constructed combination as another input.
we then perform linear regression on these samples, each containing (28x28)! features.
is it not the case that this would provide optimal accuracy?
thanks
you're describing polynomial regression.
in principle, yes, you can approximate every continuous function (on a closed interval) up to some epsilon with a polynomial (stones-weierstrass-theorem). there's a fun paper which compares polynomial regression to other methods (including MLPs) on a number of small datasets, and polynomial regression often comes out on top. (again, datasets were relatively small, which might have favoured non-NN methods... interesting to think about... but the people who wrote the paper might actually have sucked at training the neural nets and simply done it wrong)
ok, bad news is, i can't find the paper anymore.
good news is, i found this other really interesting paper: https://arxiv.org/pdf/1806.06850.pdf (still reading it)
regarding the number of combinations of inputs... that's asking which degree we choose for our polynomial. and even with a single feature you can generate an infinite number of degrees, 1, x, x², x³... one problem might be the number of parameters you need. with 784 inputs you already get roughly 80 million parameters for a polynomial of 3rd degree (if i'm not mistaken). but to be honest, i have never really experimented with it.
|
cool, thanks
read the whole thing
I get that they aren't really focusing on feature selection, but I also think they don't realize just how powerful the implications of these ideas in combination with new feature selection algorithms could be. In fact, I think feature selection will be the focus of the future. I mean the real point of dropout and why it's so effective is because it helps promote useful feature selection within the neurons of your NN. And CNNs, what are they? They are feature selectors.
So yeah, with the 28*28 image example - 80 million combinations of 3 is a lot, but if we are creating relevant combinations we would only ever need to look at data points that connect to each other. So 784 choose 3 becomes more like (9*8*784) = 56844 useful combinations, so it's not nearly as absurd. And that's without even doing any real analysis.
Hell, I even just tried some random stuff like creating a correlation matrix with the inputs, then cutting it off at some arbitrary threshold, and then using it as a graph with edge weights of 1 or 0. I then just started finding cliques, found a shitload of them, and used those cliques as my features. I then did linear regression from those features, and I got 99% on the digit database. That database is stupid though, the images are awful.
|
Northern Ireland26764 Posts
Must say thanks a lot to the OP contributors and posters in this thread. Trying to retrain at present and as much as people say ‘it’s on the internet just learn it’ I’ve always been overwhelmed by sheer amount of information, this really helps narrow it down somewhat
|
|
|
|
|
|
EPT
