I’ve recently been doing some work in R, and using the foreach package for some easy parallelization. I’d like to be able to spit out logging messages throughout my code, but the print lines always disappear when they are in the code executed by the %dopar% operator. So I’ve been working on trying to come up with a nice way to still capture/print those logging messages without having to change much of my existing code. I think my ultimate goal would be to write another binary operator which I can use as a drop-in replacement for %dopar%, which uses dopar under the covers, and which gets my print lines to the console. I haven’t quite gotten to that point yet, but I might be close-ish, and, either way, what I’ve got seems like it could either be useful for others, or something others could suggest how to fix.
My Setup
Following this post on closures in R, I set up most of my code as a bunch of ‘objects’ (environments). For example, I construct a logger object by calling the following function:
get.logger <- function() {
logger <- new.env()
logger$message <- function(str) {
print(paste(date(), str))
}
environment(logger$message) <- logger
logger
}
In sort of my ‘main’ code, I then do something like logger <- get.logger(), and then pass this logger around to my other objects which do more heavy lifting. For example, I might have a function:
guy.using.logger <- function(lgr) {
guy <- new.env()
guy$.logger <- lgr
guy$do.stuff <- function() {
foreach(i=1:10,
.inorder=TRUE,
.export='.logger',
.init=c(),
.combine=function(left, right) { c(left, right) }) %dopar% {
.logger$message(paste('Working on piece', i))
i^2
}
}
environment(guy$do.stuff) <- guy
guy
}
And then I’d have something like the following in my main code (in addition to setting up the parallel backend and such):
logger <- get.logger() gul <- guy.using.logger(logger) gul$do.stuff()
That final line return a vector with 10 elements, the squares of the integers 1 through 10, as expected. However, the thing I’m trying to overcome is that none of the expected print lines show up on the console.
Misguided Quest
I admit that this may be somewhat of a misguided quest. Changing the %dopar% to simply %do%, all of the print lines show up. And maybe I shouldn’t be doing objects the way I have them. I like to think an argument could be made for still trying to make my logging work within %dopar%. I’m not going to try to make an argument though. If you think it’s a useless mission, go find something else to read – there’s plenty of other interesting things online. Or stick around anyway, perhaps you’ll find something interesting here, or be able to help understand whatever it is I’m missing to make it work the way I want.
Buffering Messages
The basic idea I’ve been working with is to replace the logger with one that doesn’t actually print lines, but stores them up so they can be retrieved and printed later. So, for starters, I made a buffering logger:
get.buffering.logger <- function() {
logger <- new.env()
logger$.log <- c()
logger$message <- function(str) {
.log <<- c(.log, paste(date(), str))
}
logger$get.log <- function() {
rv <- .log
.log <<- c()
rv
}
environment(logger$message) <- logger
environment(logger$get.log) <- logger
logger
}
Now, if I just make logger <- get.buffering.logger() and pass that to guy.using.logger, I can’t successfully ask for the messages after a call to do.stuff(). My guess is that the issue is the lack of thread-safety with vector/c(). This isn’t too upsetting, because I don’t really want to be buffering my messages all that long anyway. If I return the logger’s buffered messages as part of the results at the end of the expression evaluated in %dopar%, then in the .combine of the foreach, I could print those messages there. More explicitly, I could do:
complicated.guy.using.logger <- function(lgr) {
guy <- new.env()
guy$.logger <- lgr
guy$do.stuff <- function() {
combiner <- function(left, right) {
cat(right$msgs, sep='\n')
c(left, right$ans)
}
foreach(i=1:10,
.inorder=TRUE,
.export='.logger',
.init=c(),
.combine=combiner) %dopar% {
.logger$message(paste('Working on piece', i))
list(ans=i^2, msgs=.logger$get.log())
}
}
environment(guy$do.stuff) <- guy
guy
}
logger <- get.buffering.logger()
cgul <- complicated.guy.using.logger(logger)
cgul$do.stuff()
And I get all my print lines, and the same return value from the %dopar% call (the vector of the first ten squares) as before. Admittedly, I’m a little surprised that this seems to work, now that I think about it. The shared logger is buffering messages for several threads, which seems sort of troublesome. I guess they’re kept separate enough by the threads, or I’ve just gotten lucky the twice I’ve run the code above.
Jazzing Things Up
If I only had one or two foreach calls in my code, I’d probably just fix them as above and be content enough (if the last thread-safety concerns calmed down). However, there’s clearly room for some automation here. What has to happen?
- The expression evaluated by %dopar% needs to return whatever it did before, as well as the accumulated messages during the execution.
- The combine function needs to return what it did before, and print out the logging messages.
The easy part, from my perspective, is the second part. Let’s assume we can solve number 1, as we did in the previous section. That is, the new block will return a list, with an ‘ans’ element which has whatever was returned previously, and a ‘msgs’ element. Recall that foreach() actually returns an object of class foreach. Using summary() to get a sense of that object, it seems easy enough to write a function which takes a foreach object and returns a new foreach object which is basically the same as the original, but has a modified .combine function. Let’s try
jazz.fe <- function(fe) {
r.fe <- fe
r.fe$combineInfo$fun <- function(left, right) {
cat(right$msgs, sep='\n')
fe$combineInfo$fun(left, right$ans)
}
r.fe
}
We need a similar function which modifies the expression to be evaluated (the block of code on the right of %dopar%), so that the return values are manipulated as described above. I messed around a little bit with passing unevaluated expressions around, and came up with the following:
jazz.ex <- function(ex) {
parse(text=c('r<-', deparse(ex), 'list(msgs=.logger$get.log(), ans=r)'))
}
And then these bits can be used as in the following guy:
jazzed.guy.using.logger <- function(lgr) {
guy <- new.env()
guy$.logger <- lgr
guy$do.stuff <- function() {
jazz.fe(foreach(i=1:10,
.inorder=TRUE,
.export='.logger',
.init=c(),
.combine=function(left, right) { c(left, right) })) %dopar% eval(jazz.ex(quote({
.logger$message(paste('Working on piece', i))
i^2
})))
}
environment(guy$do.stuff) <- guy
guy
}
This code should be compared with the very first guy.using.logger. The only difference between the two is that we wrapped the foreach in a function call, and also wrapped the expression in… a few calls. The ultimate goal of a drop-in %dopar% replacement is tantalizingly close. If all I need to do is call some function on the foreach object, and some other function on the expression, and then I can run %dopar%, that’s easy:
'%jdp%' <- function(fe, ex) {
jazz.fe(fe) %dopar% eval(jazz.ex(ex))
}
And then I could just do
failing.ultimate.guy.using.logger <- function(lgr) {
guy <- new.env()
guy$.logger <- lgr
guy$do.stuff <- function() {
foreach(i=1:10,
.inorder=TRUE,
.export='.logger',
.init=c(),
.combine=function(left, right) { c(left, right) }) %jdp% quote({
.logger$message(paste('Working on piece', i))
i^2
})
}
environment(guy$do.stuff) <- guy
guy
}
logger <- get.buffering.logger()
fgul <- failing.ultimate.guy.using.logger(logger)
fgul$do.stuff()
No dice:
> fgul$do.stuff() Error in e$fun(obj, substitute(ex), parent.frame(), e$data) : unable to find variable ".logger"
Rats. I haven’t yet found the right combination of quote, eval, substitute, … to make this work, and I’ve tried several. I’ve been reading about environments and frames and lexical scoping and closures, and still haven’t gotten it right. If I put the definition of %jdp% in the do.stuff function, it actually works. But that’s ugly. Nothing else I’ve come up with works and involves as few changes as wrapping the two arguments to the %dopar% operator, as in the jazzed.guy.using.logger above.
So, if anybody’s got any suggestions, just let me know. In the mean time, I’ll either poke around in the %dopar% source for inspiration, or move on to other things.
and that this implies 
) I remembered doing while I was reading Hardy & Wright more recently, and decided to see if I could get the inequalities right again. It took a few minutes, but I think I got there.
). And then I realized I didn’t really want to make all of those little pictures, that the fonts wouldn’t match, and that I’d probably still have resolution issues (I couldn’t find an easy way to make, say, svgs, from tex).
, as a functor of manifolds 
and 
. That’s perhaps a bit ambitious. I’ll only be thinking about 
, a disjoint union of closed disks, and I’ll actually fix 
, ranging over real vector spaces.
to something I’ll maybe denote 
. This is componentwise affine (linear followed by translation) maps 
whose restriction to 
-balls around the 0s is an embedding. I may use 
, and write 
. And I’ll actually send everything to spectra, instead of topological spaces, via 
.
, having fixed an 
and 
to give 
.
(i.e., 
). Then embeddings are just configuration spaces, 
. I’ve talked before about a homotopy limit model in this case, but let me remind you about it.
. Objects will be non-trivial partitions of 
, and I’ll probably denote them 
, perhaps writing 
. Non-trivial means that some equivalence class is larger than a singleton. I’ll write 
if 
, meaning that whenever 
, then 
.
and call “non-locally constant” maps. So 
is the set (space) of maps 
such that there is an 
. Equivalently, maps which don’t factor through 
.
(i.e., 
). Then the space of embeddings is homotopy equivalent to the space of injective linear maps. You can obtain a homotopy limit model in this case that looks strikingly similar to the previous case. Namely, you set up a category of “linear” partitions (equivalent to modding out by a linear subspace), and take the holim of the non-locally constant maps functor, as before.
, and objects in the category will be vector spaces 
with non-zero linear maps 
. Morphisms from 
to 
will be surjective linear maps 
with 
. You might think of the objects as an abstract partition (
, which then determines a partition of 
. Arone has shown
, 
, then
denotes the join. For categories, 
is 
, obtained by adding a new initial object to each of 
and 
, taking the product, and removing the initial object of the result.
. The idea is to line up the squares:
.
, linear 
. To tidy up a little bit, I’ll denote this category 
, for join. The functor takes an object as above and returns the set of componentwise affine maps 
such that either (a) the map is non-constant on some component, (b) when restricted to the image of 
, the map is non-locally constant with respect to 
, then we’d have map 
. It’d be really swell if this map was modelled by a map 
of the categories we are taking homotopy limits over. But that’s not going to happen. What can go wrong? Non-trivial partitions of 
, may become trivial, and thus not part of the category. This is, essentially, because several components of 
paired with some 0 vector spaces?
by 
, say. That should be enough of a description for now.
, to be a partition of a disjoint union of affine spaces, such that equivalence classes contain components. That is, everybody that’s in the same component is in the same equivalence class. Given a 
the underlying component-wise affine space. The data that determines a 
(the set of components), a partition of 
(non-negative integers). With this information, 
.
from 
, denoted 
, I will mean an affine map 
so that the “affine closure” of the partition 
is coarser than 
and 
(partition of possibly distinct finite sets), a refinement just means 
.
. The objects will be pairs of: a CAP, 
(subsequently denoted 
). A morphism from 
will be a refinement 
. This should look familiar to the 
. We’d really like to be able to say
, and we know the homotopy limit over 
has the right homotopy type. This is our goal.
, the Grothendieck construction is a category, which I’ll denote 
, whose objects are pairs 
where 
. Morphisms 
are morphisms 
such that 
. Of course, my functors are all contravariant as defined, so you have to mess about getting your arrows right. Best done in private.
can be written as a Grothendieck construction. Actually, it can in a few ways. The obvious way is to set 
to be the category of CAPs 
. You find that 
.
be the category of CAPs 
. Now the functor you need is not all non-locally constant maps to 
. You might denote this set 
. I’m tired of all the notation, so let me let 
denote this non-locally constant lifts functor. We have, then 
.
for 
. Notice that it is actually a functor from 
, we have 
.
has objects the morphisms of 
to 
are commuting squares
and 
are contravariant functors from 
is a covariant functor from 
. One can show that
. Between them lies a category I’ll denote 
, consisting of arrows 
with 
, 
. Morphisms are “twisted” commuting squares, as they should be, as part of the twisted arrow category. One can reduce the holim over 
to one over 
.
, the over-category 
is contractible. In fact, this result seems to rely very little on our particular 
.
, we have
, is equivalent to the right Kan extension of it’s restriction to a functor from 
. Now remember 
is a CAP with a function to 
. If this function is locally constant (all objects within an equivalence class get sent to the same point), then you sort of replace 
is non-locally constant, and so 
.
to space 
is a way to assign points in 
, and asking about the maps from 
, and repeating the process. Typically the spaces 
and 
, then we might denote the embeddings of 
. This is then a function itself – a function of two variables, 
, for example) what happens to the output values (
) if the input value (
) is changed slightly? If we know about the value at a particular point (
), what can be said about values nearby (
, say)? The answers to these questions lie with the derivative, and its “higher” analogues (the derivative of the derivative, and so on). If one knows about the derivatives of a function at a point, one can create “polynomial” approximations to the function, near that point.
for the “decimal part” of a real number, defined as 
, 
being the largest integer no bigger than 
is irrational. Let 
for 
. These points partition the interval ![[0,1]](http://s0.wp.com/latex.php?latex=%5B0%2C1%5D&bg=ffffff&fg=333333&s=0 "[0,1]")
into 
points, the 
and 
comes in when there are 
points, and those two points are still neighbors when there are 
points, then the age of that interval, at stage 
(plus 1, if you want, it doesn’t matter).
or so (basically 1 minus the golden ratio, nice and irrational). I’ve set up my notebook so that points travel from 0, at the top of the circle, clockwise, because that’s how it was done in the papers I was reading, and I thought it’d be less confusing. So here’s the starting picture, when there’s just the points 0 and 
:
, which has the shape of a 
(
, such that for almost all 
, if the continued fraction representation of ![[0;a_1,a_2,\ldots]](http://s0.wp.com/latex.php?latex=%5B0%3Ba_1%2Ca_2%2C%5Cldots%5D&bg=ffffff&fg=333333&s=0 "[0;a_1,a_2,\ldots]")
, then ![\sqrt[n]{a_1\cdots a_n}\to K](http://s0.wp.com/latex.php?latex=%5Csqrt%5Bn%5D%7Ba_1%5Ccdots+a_n%7D%5Cto+K&bg=ffffff&fg=333333&s=0 "\sqrt[n]{a_1\cdots a_n}\to K")
as 
.
, according to Wikipedia anyway. An exact expression for 
. I’ll take 
, and we let 
be the largest integer less than 
, denoted 
, and we know that 
. That leaves a little bit over, which I’ll denote 
. We have 
, the first step in our continued fraction. You now iterate this process, looking at 
, setting 
and 
, and obtaining
(all positive integers) and 
(all in ![\alpha=[0;a_1,a_2,\ldots]](http://s0.wp.com/latex.php?latex=%5Calpha%3D%5B0%3Ba_1%2Ca_2%2C%5Cldots%5D&bg=ffffff&fg=333333&s=0 "\alpha=[0;a_1,a_2,\ldots]")
. That initial zero is just becuase I started with 
, if you started with an 
outside ![[a_1,a_2,\ldots]](http://s0.wp.com/latex.php?latex=%5Ba_1%2Ca_2%2C%5Cldots%5D&bg=ffffff&fg=333333&s=0 "[a_1,a_2,\ldots]")
, dropping the initial zero.
, a sequence of values 
. Restated, we have functions 
, for collections of 
s and 
s. That is, given 
and 
, all positive integers, what does the set of all ![\alpha=[a_1,a_2,\ldots]](http://s0.wp.com/latex.php?latex=%5Calpha%3D%5Ba_1%2Ca_2%2C%5Cldots%5D&bg=ffffff&fg=333333&s=0 "\alpha=[a_1,a_2,\ldots]")
such that 
for 
look like?
look like? Well, 
means that 
. Which is to say, 
, or 
. So the graph of 
is a step function, taking the value ![(1/(k+1),1/k]](http://s0.wp.com/latex.php?latex=%281%2F%28k%2B1%29%2C1%2Fk%5D&bg=ffffff&fg=333333&s=0 "(1/(k+1),1/k]")
. I picture this as a staircase descending as you move to the right. Of course, the stairs are pretty narrow on the left…
is ![(1/4,1/3]](http://s0.wp.com/latex.php?latex=%281%2F4%2C1%2F3%5D&bg=ffffff&fg=333333&s=0 "(1/4,1/3]")
, the third stair from the right.
. What does 
look like? Well, in finding the continued fraction for 
then 
, or 
. So we find that for any 
and 
have 
. That is, in each of the “stairs” from the discussion of 
comprises, in the ascending staircases in each interval 
and 
as above, the set of all 
, 
, and 
. I’ll denote the set of appropriate 
, and the measure of that set by 
.
to be the measure of the set of all 
, and found that
. I’ll let you read about it in Khinchin’s book. Anyway, in 1928, Kuz’min came along and found a generalization. Kuz’min’s theorem gives a more precise result than Gauss’ (at least, as stated above). Namely:
and 
, such that
have nothing to do with eachother, so the measure that we are calculating is of a completely different set with each 
is a function from positive integers to positive reals such that 
for some positive constants 
and 
, then for almost every 
, simply messing with the equality using log and exponent rules.
if 
and 0 otherwise. Then the limit on the left of the equality in Khinchin’s theorem should be interpreted as the density of 
among the values of the 
we find that approximately 42% of the terms (Khinchin calls them “elements”) in the continued fraction for just about any ![[a_1,\ldots]](http://s0.wp.com/latex.php?latex=%5Ba_1%2C%5Cldots%5D&bg=ffffff&fg=333333&s=0 "[a_1,\ldots]")
is the fraction ![[a_1,\ldots,a_n]](http://s0.wp.com/latex.php?latex=%5Ba_1%2C%5Cldots%2Ca_n%5D&bg=ffffff&fg=333333&s=0 "[a_1,\ldots,a_n]")
. As this is a finite continued fraction, it represents a rational number, and it is typically denoted 
(relatively prime). The growth of the denominators in the sequence of convergents in a continued fraction is a fairly worthwhile question. Using some elementary relationships among the 
and 
, you can find that 
. The interesting theorem I have in mind is another “there is a constant” theorem:![\alpha=[a_1,\ldots]](http://s0.wp.com/latex.php?latex=%5Calpha%3D%5Ba_1%2C%5Cldots%5D&bg=ffffff&fg=333333&s=0 "\alpha=[a_1,\ldots]")
,![\displaystyle \lim_{n\to \infty}\sqrt[n]{q_n}=e^{\pi^2/(12\ln 2)}.](http://s0.wp.com/latex.php?latex=%5Cdisplaystyle+%5Clim_%7Bn%5Cto+%5Cinfty%7D%5Csqrt%5Bn%5D%7Bq_n%7D%3De%5E%7B%5Cpi%5E2%2F%2812%5Cln+2%29%7D.&bg=ffffff&fg=333333&s=0 "\displaystyle \lim_{n\to \infty}\sqrt[n]{q_n}=e^{\pi^2/(12\ln 2)}.")
