?

Log in

No account? Create an account
smart migrator - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

smart migrator [Jan. 11th, 2005|09:15 pm]
Brad Fitzpatrick
I just wrote a database migration script which, after each chunk of data moved, asks the load balancer (Perlbal) the free user queue depth. If more than 10 (less is just noise), it sleeps a second and asks again. Only once the queues are empty does it migrate more data.

End result: it moves data as fast as it can, without affecting page response times.

1432000/81000000 = 1.77%
1433000/81000000 = 1.77%
1434000/81000000 = 1.77%
wait_back = 10
1435000/81000000 = 1.77%
1436000/81000000 = 1.77%
....
1450000/81000000 = 1.79%
1451000/81000000 = 1.79%
1452000/81000000 = 1.79%
wait_back = 35
1453000/81000000 = 1.79%
1454000/81000000 = 1.80%
1455000/81000000 = 1.80%
...
1470000/81000000 = 1.81%
1471000/81000000 = 1.82%
1472000/81000000 = 1.82%
wait_back = 14
wait_back = 10
wait_back = 67
wait_back = 68
wait_back = 50
wait_back = 67
wait_back = 91
wait_back = 72
1473000/81000000 = 1.82%
1474000/81000000 = 1.82%
1475000/81000000 = 1.82%

I've been meaning to make a generic wrapper for any utility, where the wrapper parent watches the load balancers, and the child does its work at full speed, but the parent will occasionally SIGSTOP/SIGCONT it...... haven't got around to that yet. The generic wrapper could even have another pluggable-child as its rate-limit determiner, so anybody could use it.
LinkReply

Comments:
From: nathanmock
2005-01-12 05:30 am (UTC)
Did you mean 'affecting'?
(Reply) (Thread)
[User Picture]From: brad
2005-01-12 05:36 am (UTC)
Whatcha talking about? My post said a-for-action-affecting all along! :P

Think I wrote e-for-"endresult"-effecting? Think I need silly mnemonics to help me remember everything?
(Reply) (Parent) (Thread)
From: nathanmock
2005-01-12 05:46 am (UTC)
Hehe. Let's go back to first grade, b-for-rad-Brad! :)
(Reply) (Parent) (Thread)
From: evan
2005-01-12 10:28 am (UTC)
so the funny thing about these words is they have secret meanings when they switch forms.

i tend to think affect/verb = action and effect/noun = result, but there's also affect/noun = disposition and effect/verb = to cause to happen.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2005-01-12 10:34 am (UTC)
You've told me about effect/verb before, but I didn't know affect/noun == disposition.

Wanna go to Mexico?
(Reply) (Parent) (Thread)