The Big Programming Thread - Page 981

Prev 1 979 980 981 982 983 1032 Next

Thread Rules
1. This is not a "do my homework for me" thread. If you have specific questions, ask, but don't post an assignment or homework problem and expect an exact solution.
2. No recruiting for your cockamamie projects (you won't replace facebook with 3 dudes you found on the internet and $20)
3. If you can't articulate why a language is bad, don't start slinging shit about it. Just remember that nothing is worse than making CSS IE6 compatible.
4. Use [code] tags to format code blocks.

WarSame

Canada1950 Posts

December 12 2018 01:34 GMT

#19601

I did something similar in my Flask project.

Dockerfile

Docker compose

IIRC the main thing was in the Docker compose file, under web: volumes. That automatically updates from my /flask directory in the project to the /app directory in the container.

I did notice a problem that I would have to run into the same code twice before it took effect, so there should be a better way.

Manit0u

Poland17625 Posts

December 12 2018 02:36 GMT

#19602

On December 12 2018 10:34 WarSame wrote:
I did something similar in my Flask project.

Dockerfile

Docker compose

IIRC the main thing was in the Docker compose file, under web: volumes. That automatically updates from my /flask directory in the project to the /app directory in the container.

I did notice a problem that I would have to run into the same code twice before it took effect, so there should be a better way.

Can it be because python creates those intermediate .pyc files?

I'm not entirely sure how flask creates cache and stuff. PHP server, rails, mongrel, sinatra etc. all pick up changes to files immediately (unless you turn off their file watcher).

WarSame

Canada1950 Posts

December 12 2018 14:59 GMT

#19603

I am not sure, I just accepted it

R1CH

Netherlands10341 Posts

December 12 2018 16:17 GMT

#19604

Most Python app servers will need relaunching to detect source code changes.

Manit0u

Poland17625 Posts

December 14 2018 10:00 GMT

#19605

So, I completely forgot to post Dockerfile and docker-compose...
Here they are:

+ Show Spoiler [Dockerfile] +


FROM ruby:2.3.6

# Install APT dependencies
RUN apt-get update && \
  DEBIAN_FRONTEND=noninteractive apt-get install -y \
    cmake \
    curl \
    git \
    mysql-client \
    openssh-client \
    sqlite3 \
    wget

ENV LC_ALL=C.UTF-8

ARG TMP_DIR=/tmp/
RUN mkdir -p ${TMP_DIR} && chmod 1777 ${TMP_DIR}

# Zeromq
ARG LINUX_LIBZMQ_VERSION=4.0.5
RUN ZMQ_TMP_DIR=${TMP_DIR}/zmq/ && \
      mkdir -p ${ZMQ_TMP_DIR} && \
      wget [url=http://download.zeromq.org/zeromq-${LINUX_LIBZMQ_VERSION}.tar.gz]http://download.zeromq.org/zeromq-${LINUX_LIBZMQ_VERSION}.tar.gz[/url] \
        -O ${ZMQ_TMP_DIR}/zeromq-${LINUX_LIBZMQ_VERSION}.tar.gz && \
      tar -xvzf ${ZMQ_TMP_DIR}/zeromq-${LINUX_LIBZMQ_VERSION}.tar.gz -C ${ZMQ_TMP_DIR} && \
      cd ${ZMQ_TMP_DIR}/zeromq-${LINUX_LIBZMQ_VERSION} && \
      ./configure && make && make install && ldconfig && \
      test ${ZMQ_TMP_DIR} && test -d ${ZMQ_TMP_DIR} && rm -rf ${ZMQ_TMP_DIR}

# Mongrel
ARG LINUX_MONGREL_VERSION=1.9.2
RUN MONGREL_TMP_DIR=${TMP_DIR}/mongrel/ && \
      mkdir -p ${MONGREL_TMP_DIR} && \
      wget [url=https://github.com/mongrel2/mongrel2/releases/download/${LINUX_MONGREL_VERSION}/mongrel2-v${LINUX_MONGREL_VERSION}.tar.bz2]https://github.com/mongrel2/mongrel2/releases/download/${LINUX_MONGREL_VERSION}/mongrel2-v${LINUX_MONGREL_VERSION}.tar.bz2[/url] \
        -O ${MONGREL_TMP_DIR}/mongrel2-v${LINUX_MONGREL_VERSION}.tar.bz2 && \
	tar -xvjf ${MONGREL_TMP_DIR}/mongrel2-v${LINUX_MONGREL_VERSION}.tar.bz2 -C ${MONGREL_TMP_DIR} && \
	cd ${MONGREL_TMP_DIR}/mongrel2-v${LINUX_MONGREL_VERSION}/ && \
	make -C ${MONGREL_TMP_DIR}/mongrel2-v${LINUX_MONGREL_VERSION} && \
	make -C ${MONGREL_TMP_DIR}/mongrel2-v${LINUX_MONGREL_VERSION} install && \
      test ${MONGREL_TMP_DIR} && test -d ${MONGREL_TMP_DIR} && rm -rf ${MONGREL_TMP_DIR}

ARG USER_ID=1000
ARG GROUP_ID=1000
RUN groupadd -r dev --gid ${GROUP_ID} && \
      useradd --no-log-init --system --create-home --uid ${USER_ID} --gid ${GROUP_ID} dev
USER dev
RUN mkdir -p /home/dev/.ssh

ENV LINES=${LINES}
ENV COLUMNS=${COLUMNS}
WORKDIR /home/dev/app
ENTRYPOINT /bin/bash

+ Show Spoiler [docker-compose] +


version: '3.5'

services:
  engine:
    build:
      context: "."
      dockerfile: Dockerfile
      args:
        USER_ID: ${USER_ID}
        GROUP_ID: ${GROUP_ID}
    environment:
      LINES: ${LINES}
      COLUMNS: ${COLUMNS}
      LANG: en_US.UTF-8
      DOCKER_LOCAL: 'true'
    entrypoint: /bin/bash
    command: "-c 'trap : TERM INT; sleep infinity & wait'"
    ports:
      - "6767:6767"
    volumes:
      - "${SSH_KEY_PATH}:/home/dev/.ssh/id_rsa:ro"
      - "${SSH_PUBLIC_KEY_PATH}:/home/dev/.ssh/id_rsa.pub:ro"
      - "${KNOWN_HOSTS_PATH}:/home/dev/.ssh/known_hosts:ro"
      - ./:/home/dev/app/:cached
    networks:
      - infrastructure

networks:
  infrastructure:
    external: true

Edit:

Bonus

Manit0u

Poland17625 Posts

December 14 2018 16:03 GMT

#19606

Holy crap! Today, after 2 years of working with it I've discovered something new about Ruby. You can actually create this kind of hash:

{ nil => 'some value' }

This is fucking absurd...

Deleted User 3420

24492 Posts

December 15 2018 00:04 GMT

#19607

I was checking things that are using up a lot of space on my computer.

My AMD folder is 10gb
and frankly i don't even remember installing any updates or anything after I initially installed my video drivers

what in the world is going on there? is it like auto updating and then just saving the old folders for eternity or something?

Manit0u

Poland17625 Posts

December 15 2018 10:46 GMT

#19608

On December 15 2018 09:04 travis wrote:
I was checking things that are using up a lot of space on my computer.

My AMD folder is 10gb
and frankly i don't even remember installing any updates or anything after I initially installed my video drivers

what in the world is going on there? is it like auto updating and then just saving the old folders for eternity or something?

Usually when you download drivers and update them it unpacks them to intermediate folder first and then leaves them there.

Excludos

Norway8231 Posts

December 15 2018 12:04 GMT

#19609

On December 15 2018 19:46 Manit0u wrote:

Show nested quote +

Usually when you download drivers and update them it unpacks them to intermediate folder first and then leaves them there.

Always get rid of temporary files when updating or installing a client.

That said I'm having a similar issue right now myself. The whole pipe is a bit complicated, but the gist is that I'm ending up with some software being run on a VM at regular intervals, but unable to delete itself once it's done. It's also unable to delete any previous versions of itself as every run has a different generated user, and you don't have access to other user's files.

This is all done through Azure Data Factory as well, so I have very limited options to do anything with. I can't, for instance, just log onto the VM and delete the files manually either (Not that I'd want to, this is suppose to work without human interaction).

mantequilla

Turkey781 Posts

December 15 2018 21:20 GMT

#19610

I use a tiny program called SpaceSniffer time to time, it visualizes what is wasting space pretty good. Sometimes I notice those kind of weird things otherwise I'd overlook.

mantequilla

Turkey781 Posts

December 17 2018 09:56 GMT

#19611

umm, a weird thing I noticed when trying to do this is,

volumes are mapped to host machine. Since I work on windows, there is an intermediate linux vm running inside hyper-v. The real host for my docker containers is that linux vm

So any volume I create is not in my host machine but in that virtual machine. So I am not able to do this :/

zatic

Zurich15359 Posts

December 17 2018 10:27 GMT

#19612

Is there an elegant way to join random date slices into fewer, longer date slices?

I have X startdate-enddate periods that can range from 1 day to many years. I want to join any overlapping periods and remove periods that lie within another period. In the end I want one list of periods with no overlaps. I can come up with some complicated semi-brutforce:

sort by start date
go over every period
search back through result from current period for overlaps
delete / join overlaps
add to / replace in result

Maybe there is no good way but it seems like such a straightforward thing to do and I feel stupid for not finding a more elegant way.

slmw

Finland233 Posts

December 17 2018 10:58 GMT

#19613

Sort period by start date, then:
1. Find the first (next) period and save its start date and end date into some variables as "current period"
2. For every following period that starts before the "current period" ends, update it accordingly (move end date if the following period ends later than "current period")
3. Add "current period" to result and jump to (1.).

Acrofales

Spain18199 Posts

December 17 2018 12:26 GMT

#19614

On December 17 2018 19:27 zatic wrote:
Is there an elegant way to join random date slices into fewer, longer date slices?

I have X startdate-enddate periods that can range from 1 day to many years. I want to join any overlapping periods and remove periods that lie within another period. In the end I want one list of periods with no overlaps. I can come up with some complicated semi-brutforce:

sort by start date
go over every period
search back through result from current period for overlaps
delete / join overlaps
add to / replace in result

Maybe there is no good way but it seems like such a straightforward thing to do and I feel stupid for not finding a more elegant way.

Depending on your data this may not be possible (or rather, you may end up with a single "period", which may or may not be what you want).

slmw gave a good O(n log n) algorithm for doing it (well, O(n) for the interesting bit. The worst case complexity is from sorting).

AKnopf

Germany259 Posts

December 17 2018 15:12 GMT

#19615

Edit: Forget it. I'm stupid

enigmaticcam

United States280 Posts

December 17 2018 18:04 GMT

#19616

You can also do it in SQL pretty easily. Something like this:

--Get list of date overlaps
select *
from #DateTable a
inner join #DateTable b
	on b.StartDate <= a.EndDate --match on overlap
	and b.EndDate >= a.StartDate --match on overlap
	and b.Id > a.Id --Ensure we get an overlap just once
	and (b.StartDate != a.StartDate or b.EndDate != a.EndDate) -- If two dates are both exact, might want to handle that separately

Once you have the list, you'll just need to do an update and determine how to remove the overlaps.

Manit0u

Poland17625 Posts

December 18 2018 08:40 GMT

#19617

On December 18 2018 03:04 enigmaticcam wrote:
You can also do it in SQL pretty easily. Something like this:

--Get list of date overlaps
select *
from #DateTable a
inner join #DateTable b
	on b.StartDate <= a.EndDate --match on overlap
	and b.EndDate >= a.StartDate --match on overlap
	and b.Id > a.Id --Ensure we get an overlap just once
	and (b.StartDate != a.StartDate or b.EndDate != a.EndDate) -- If two dates are both exact, might want to handle that separately

Once you have the list, you'll just need to do an update and determine how to remove the overlaps.

You can use CASE WHEN to update it

Edit:

Also this...


date_ranges_1 = [
  Sat, 15 Dec 2018..Mon, 17 Dec 2018, 
  Fri, 14 Dec 2018..Mon, 17 Dec 2018, 
  Thu, 13 Dec 2018..Mon, 17 Dec 2018, 
  Wed, 12 Dec 2018..Mon, 17 Dec 2018, 
  Tue, 11 Dec 2018..Mon, 17 Dec 2018
] 
date_ranges_2 = [
  Mon, 17 Dec 2018..Thu, 20 Dec 2018, 
  Sun, 16 Dec 2018..Thu, 20 Dec 2018, 
  Sat, 15 Dec 2018..Thu, 20 Dec 2018, 
  Wed, 19 Dec 2018..Sun, 23 Dec 2018, 
  Thu, 20 Dec 2018..Sun, 23 Dec 2018
]

res = date_ranges_1.map do |a|
  latest_ending = date_ranges_2.select { |b| a.include?(b.first) }.max_by { |b| b.last }

  if latest_ending.nil?
    a
  else
    a.last < latest_ending.last ? (a.first..latest_ending.last) : a
  end
end

res => [
  Sat, 15 Dec 2018..Thu, 20 Dec 2018, 
  Fri, 14 Dec 2018..Thu, 20 Dec 2018, 
  Thu, 13 Dec 2018..Thu, 20 Dec 2018, 
  Wed, 12 Dec 2018..Thu, 20 Dec 2018, 
  Tue, 11 Dec 2018..Thu, 20 Dec 2018
]

Is something like that you want?

You can also do something like that:


date_ranges_1.map do |a|
  earliest_starting = date_ranges_2.select { |b| a.include?(b.last) }.min_by { |b| b.first } || a
  latest_ending = date_ranges_2.select { |b| a.include?(b.first) }.max_by { |b| b.last } || a

  earliest_starting.first..latest_ending.last
end

Acrofales

Spain18199 Posts

December 18 2018 10:29 GMT

#19618

On December 18 2018 03:04 enigmaticcam wrote:
You can also do it in SQL pretty easily. Something like this:

--Get list of date overlaps
select *
from #DateTable a
inner join #DateTable b
	on b.StartDate <= a.EndDate --match on overlap
	and b.EndDate >= a.StartDate --match on overlap
	and b.Id > a.Id --Ensure we get an overlap just once
	and (b.StartDate != a.StartDate or b.EndDate != a.EndDate) -- If two dates are both exact, might want to handle that separately

Once you have the list, you'll just need to do an update and determine how to remove the overlaps.

You obviously leave optimization there to your db engine, but it looks like the naive implementation there is O(n^2), and I'm not sure it's sargable (I guess it probably is, with separate indexes on both startdate and enddate, so it should actually also be O(n log n)). Would, in general, the SQL implementation of this really be better than just getting the sorted list from the DB and doing it in whatever language you're using?

I always wonder that when ppl try to do computation in SQL

enigmaticcam

United States280 Posts

December 18 2018 18:06 GMT

#19619

On December 18 2018 17:40 Manit0u wrote:
You can use CASE WHEN to update it

Yep! I was just being lazy and didn't bother to figure out what that would look like.

On December 18 2018 19:29 Acrofales wrote:You obviously leave optimization there to your db engine, but it looks like the naive implementation there is O(n^2), and I'm not sure it's sargable (I guess it probably is, with separate indexes on both startdate and enddate, so it should actually also be O(n log n)). Would, in general, the SQL implementation of this really be better than just getting the sorted list from the DB and doing it in whatever language you're using?

I always wonder that when ppl try to do computation in SQL

I posted it mainly for informational purposes. I depends entirely on your implementation and where your data is. If it's starting in a database and it's large with proper indexing, I think a SQL query would definitely be faster than pulling it onto a client, running local code against it, and sending back the results. But that's not always the case for everyone

bangsholt

Denmark138 Posts

December 18 2018 19:47 GMT

#19620

On December 18 2018 19:29 Acrofales wrote:
I always wonder that when ppl try to do computation in SQL

I always wonder why people want to transfer data to a client only for the client to filter and sort it.

It's almost like a SQL database has WHERE and GROUP BY

Prev 1 979 980 981 982 983 1032 Next

Please or register to reply.

The Big Programming Thread - Page 981

Completed

Ongoing

Upcoming