Behavioral Ecology Advance Access originally published online on June 19, 2006
Behavioral Ecology 2006 17(5):799-809; doi:10.1093/beheco/arl008
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Learning rules for optimal selection in a varying environment: mate choice revisited
a Centre for Behavioural Biology and Department of Mathematics, University of Bristol, University Walk, Bristol BS8 1TW, United Kingdom and b Institute of Mathematics and Computer Science, Wroc
aw University of Technology, Wybrze
e Wyspia
skiego 27, PL-50-370 Wrocaw, Poland
Address correspondence to J.M. McNamara. E-mail: john.mcnamara{at}bristol.ac.uk.
Received 1 December 2005; revised 28 April 2006; accepted 12 May 2006.
 |
ABSTRACT |
|---|
TOP
ABSTRACT
INTRODUCTIONThe quality of a chosen partner can be one of the most significant factors affecting an animal's long-term reproductive success. We investigate optimal mate choice rules in an environment where there is both local variation in the quality of potential mates within each local mating pool and spatial (or temporal) variation in the average quality of the pools themselves. In such a situation, a robust rule that works well across a variety of environments will confer a significant reproductive advantage. We formulate a full Bayesian model for updating information in such a varying environment and derive the form of the rule that maximizes expected reward in a spatially varying environment. We compare the theoretical performance of our optimal learning rule against both fixed threshold rules and simpler near-optimal learning rules and show that learning is most advantageous when both the local and environmental variances are large. We consider how optimal simple learning rules might evolve and compare their evolution with that of fixed threshold rules using genetic algorithms as minimal models of the relevant genetics. Our analysis points up the variety of ways in which a near-optimal rule can be expressed. Finally, we describe how our results extend to the case of temporally varying environments.
Key words: Bayesian model, evolution of learning rules, genetic algorithm, sequential search, sexual selection, spatially varying environment.
|
INTRODUCTION |
|---|
The quality of a chosen partner can be one of the most significant factors affecting an animal's long-term reproductive success. As a result, the performance of different mate choice rules has been, and continues to be, of central interest to behavioral ecologists (Janetos 1980
; Parker 1983; Real 1990; Simao and Todd 2002; Hutchinson and Halupka 2004; Wong and Candolin 2005). Here we investigate optimal mate choice rules in a varying environment. We focus on modeling the case where only one sex is discriminating. This restricted focus enables us to clearly set out the principles underlying the structure and evolution of robust rules and the logic of why and when learning can be advantageous. Thus, we aim to combine precise quantitative results with qualitative motivation. Moreover, we expect the intuition and insight developed to apply to more general optimal selection problems and also to carry over to more realistic game theoretic mate selection models where both sexes are choosy.
For simplicity, we refer throughout to the choosy sex as female; our analysis applies equally well when the roles are reversed (Real 1990). We assume that each individual female encounters a possibly unlimited sequence of potential mates, whose qualities are assumed to be chosen at random from a given (local) distribution, and we assume that costs of some form are associated with searching (Alatalo et al. 1988; Milinski and Bakker 1992). Under these assumptions, at any given time the female will have an acceptance threshold that will determine whether or not she accepts the current male.
Previous mate choice models have in general assumed that each female knew (i.e., was fully adapted to) the local distribution of male qualities, which in turn implicitly assumed that this local distribution remained constant from generation to generation (Janetos 1980; Parker 1983; Real 1990). For this setting, Real (1990) gives a comprehensive treatment of both sequential search rules and "best-of-n" rules with costs. He shows that a sequential search rule is optimal in terms of expected reward and will always dominate the best-of-n rule when the possible number of encounters is unlimited.
In reality, the distribution of male qualities may differ from place to place and from season to season (Svensson and Sinervo 2004). A mate choice rule that is optimal when qualities have one distribution may perform very badly when they have another (Mazalov et al. 1996; Luttbeg 2002). A female that inherits a robust rule that works well across a variety of environments will thus have a significant reproductive advantage. There may be even more advantage in an adaptive rule that learns from experience to respond optimally to the local distribution, and there is experimental evidence that female choice is affected by the quality of previously encountered males (Bakker and Milinski 1991; Milinski and Bakker 1992; Downhower and Lank 1994; Collins 1995).
Real (1990) recognized both the appropriateness of, and the difficulties inherent in, a Bayesian approach to this adaptive choice model but did not address its solution. Since then, there has been only limited progress in characterizing and evaluating the form of the optimal rule in the presence of variability and uncertainty. Apart from some simulation comparisons (Mazalov et al. 1996; Luttbeg 2002; Hutchinson and Halupka 2004), there are few clear theoretical results (Dombrovsky and Perrin 1994; Mazalov et al. 1996). This paper is the first to give a full Bayesian treatment of the optimal learning rule for mate choice in a varying environment.
Our model incorporates 2 sources of variability that may lead a female to be uncertain about the quality of a given male relative to others she might meet—variation of individual male qualities within the local pool of potential mates (residual variability) and variation of the average quality of the pools themselves, say from place to place in a given season or from season to season (environmental variability). The different types of rules allowed under our model range from rules that specify a completely fixed acceptance threshold, or rules where the acceptance threshold may change with each encounter but only in some predetermined way that is independent of the actual qualities encountered (but may, e.g., depend on the number of encounters), through to much more complex fully adaptive or learning rules where the acceptance threshold at any encounter depends crucially on the female's own past experience.
Within this framework, we are able to fully characterize and quantify the optimal (learning) rule. Perhaps more significantly, we are able to draw clear qualitative conclusions about how the relative strengths of the 2 sources of variation affect the relative success of each type of rule and also how these relative strengths affect the trade-off between the complexity of a rule and its expected reward in the presence of evolutionary selection pressures. Thus, our emphasis is on optimality rather than on heuristics (Simao and Todd 2002; Hutchinson and Halupka 2004).
More complicated models are possible, which allow for game theoretic aspects, perhaps due to both sexes being choosy or to the distribution of male qualities changing over the mating season as females remove the best males (McNamara and Collins 1990; Collins and McNamara 1993; Johnstone 1997; Bergstrom and Real 2000) or allow for courtship (Wong and Candolin 2005) or imperfect observation of male qualities, so that female choice rules need to take into account the costs and benefits of time spent in further assessment or reassessment of previously encountered males (Luttbeg 1996; Fawcett and Johnstone 2003b). We do not address these issues here.
We start by specifying the details of our model and developing a clearer intuition of the advantages and disadvantages of rules with a fixed acceptance threshold, gaining insight from the asymmetric shape of the plot of expected net gain against threshold. We formulate a full Bayesian model for updating information in a varying environment and derive the form of the corresponding optimal learning rule that maximizes expected reward in a spatially varying environment. We explore when learning is valuable—either theoretically valuable or valuable in a more practical setting that takes into account robustness and adaptability under evolutionary pressures. Thus, we first identify the conditions that might favor learning under our model by comparing the theoretical performance of our optimal learning rule against both fixed threshold rules and simpler near-optimal learning rules. We then consider how optimal simple learning rules might evolve and compare their evolution with that of fixed threshold rules using genetic algorithms (GAs) as minimal models of the relevant genetics. Our analysis points up the variety of ways in which a near-optimal rule can be expressed. Finally, we describe how our results extend to the case of temporally varying environments.
|
MODELS FOR MATE CHOICE IN A VARYING ENVIRONMENT |
|---|
Our starting point is a model similar to that of previous authors (Janetos 1980
; Parker 1983; Real 1990). Each potential mate is characterized by a value representing his quality or reproductive fitness, and this value can be immediately recognized by the female. The female encounters a sequence of males with individual fitness values x1, x2, x3, ..., where there is no limit to the potential number of encounters. She sequentially accepts or rejects each male according to his fitness value and cannot recall previously rejected males. The female stops searching if and when she finally accepts a male. We assume that individual male fitness values can be modeled as independent observations from the N(µ, 
2) distribution, that is, the normal distribution with mean µ and variance 2. For clarity of presentation, we assume the time between encounters, measured in appropriate units, has constant value 1. Finally, we assume that the female incurs a fixed positive search cost c for each time unit that elapses up to and including the time a male is finally chosen, reflecting time or energetic costs incurred. The overall net fitness gain to the female is the fitness value of the male eventually chosen minus the total search costs.
Previous authors have used this basic model to study optimal mate choice rules for females in a constant environment, where each female in each generation independently encounters males with fitness values drawn from the same N(µ, 2) distribution with known values of µ and 2. We extend this model to study optimal mate choice rules in environments where the value of µ experienced by different females can vary, either within each generation or across generations.
We now define a spatially varying environment to be one in which the value of µ varies from female to female within a given generation, according to a normal distribution with mean µ0 and variance 
The values of 2, µ0, and 
are assumed to be known and constant from generation to generation. We can gain insight into the spatially varying model by interpreting the average fitness values seen by females as being influenced by factors specific to the local environment or patch, so that µ represents a local average male fitness value. Indeed, we will sometimes speak of males "in a given location," though this is by no means the only interpretation of how this variability might occur.
We define a temporally varying environment to be one in which the value of µ is the same for all females within a given generation but now varies from generation to generation according to a normal distribution. Again we write µ0 and for the mean and variance of this distribution, and we assume that the values of 2, µ0, and are known and constant from generation to generation. The temporally varying model describes cases where the average male fitness value is essentially the same across locations in a given generation, but where this average value changes appreciably from generation to generation because of seasonal factors that affect all locations equally.
In both these models, we interpret µ0 as the long-run average male fitness value, we interpret as the environmental variance (the variance of µ about this value µ0, either within generations or across generations according to the context), and we interpret 2 as the residual variance (the variability of local male fitness values around their local average).
In many cases, quantities of interest will depend linearly on µ0 and depend on and 2 only through the value of the ratios c/
0, c/, or 0/. In the numerical examples used to illustrate our results, we will therefore mainly focus on fixed values for the parameters µ0 and c (taking µ0 = 5 and c = 0.1) and a fixed range of values for the parameters and 2 (0 = 0.5, 1, 2 and = 0.5, 1, 2), giving an appropriate range of illustrative values.
Finally, we note that many of our qualitative results remain the same if we relax the distributional assumptions to allow distributions with roughly the same shape as the normal or if we assume mortality or discounted costs (Real 1990). Similarly, the analysis is essentially unchanged if we allow the times between encounters to be continuous variables, randomly chosen from some given distribution.
|
FIXED THRESHOLD RULES IN CONSTANT AND VARYING ENVIRONMENTS |
|---|
A mate choice rule that is particularly attractive because of its simplicity is a fixed threshold rule with threshold T of the form:
 | (1) |
showed that a fixed threshold rule is optimal when the environment is constant, and the value of µ is fixed (corresponding to the case 0 = 0 in our model). Here we review the constant environment case in order to develop insight into how the performance of these fixed threshold rules is affected when the environment varies. In a constant environment, the mate choice problem is stationary in that the expected net future gain of continuing under an optimal rule is constant, independent of the current stage n and the current and previous observed male values. This follows from the fact that the values of successive males are independently drawn from the same known distribution and that recall is not allowed. Denote the expected net future gain under an optimal rule by V*. As with any similar sequential search model, an optimal female action at each stage is to accept the current prospective male if and only if the reward from mating is greater than or equal to the expected net future gain under an optimal rule. It follows that the optimal rule in a constant environment is a fixed threshold rule and that the optimal threshold T* at each stage is exactly equal to V*.
In our model, we have specifically excluded recall of previously rejected males as an option. However, because the optimal rule uses a fixed threshold, we see that in a constant environment, the recall of previously rejected potential mates would be of no benefit; a male deemed unacceptable when first encountered would have a fitness value below the fixed threshold and thus would always be unacceptable in the future.
An explicit expression for the expected overall net fitness gain, V(T), under a fixed threshold rule with threshold T, is given by the formula
 | (2) |
and 
are, respectively, the probability density function and distribution function of the N(0, 1) distribution. Here 
is the probability a randomly chosen male will exceed the threshold T and 
is the expected search time (i.e., the number of males encountered up to and including the one accepted). Figure 1 shows typical plots of V(T) for a range of c values; in each case, the graph takes value µ – c for T small, increases slowly with increasing T, and then falls steeply for larger values of T. Because T* = V* = V(T*), the values of T* and V* can be read off from the graph as the coordinates of the point of intersection with the line of unit slope through the origin.
|
To motivate the shape of the graph of V(T), note that V(T) is the difference between the expected value of the male eventually accepted and the expected cost of searching. Both of these terms are increasing functions of the threshold T. For T small, the expected value of the male accepted takes a value near µ, and the expected search time takes a value near 1. As T increases, the expected value of the male accepted also increases, initially at a faster rate than the expected search time (and hence the expected search cost), so V(T) increases. Eventually, the rate of increase of the expected search time (and cost) becomes so large that the expected cost of using a higher threshold starts to outweigh the increase in the expected value of the male accepted, until finally the costs dominate the expected return.
The important feature of this graph is its emphatically asymmetric shape; it is quite flat for values of T less than the optimal value T* but falls very steeply for values of T greater than T*. The expected overall net gain for a female using a fixed threshold less than T* is close to that under an optimal rule, even if T is appreciably less than T*. However, her expected overall net gain drops sharply if she uses a fixed threshold significantly greater than T*. We would therefore expect that selective pressure on females using a fixed threshold would lead them to err on the low side rather than the high side in choosing the threshold value.
When male values come from the N(µ, 2) distribution, V* and T* are given explicitly by
 | (3) |
for real values of x (Real 1990). Thus, the fixed threshold used by the optimal rule is of the form
 | (4) |
is a relative threshold representing how good a male has to be, relative to the known mean, to be acceptable. Note that the optimal fixed threshold T* and hence V* both increase linearly with µ for fixed c and .
Consider 2 females following optimal rules in 2 environments that differ only in the mean male fitness value, say µ1 and µ2. Although the optimal thresholds 
and 
would differ, the relative thresholds and the search time distribution would be the same, so the observed pattern of searching and acceptance would look exactly the same in both cases. However, the actual thresholds used and the expected value of the mate actually obtained would differ by an amount µ1 – µ2, reflecting the difference in the overall mean fitness values.
Moreover, if by mistake a female in an environment where the mean was µ2 used the fixed threshold 
that was optimal for an environment where the mean was µ1, then we see from the shape of the graph of V(T) that she would do only slightly worse than the optimal if µ1 was less than µ2 because then 
but she would do very much worse if µ1 was significantly greater than µ2 as then would be significantly greater than 
|
BAYESIAN MODEL FOR A VARYING ENVIRONMENT |
|---|
Now consider a female searching for a mate in a spatial or temporally varying environment. Under the varying environment model, the local male fitness values are drawn from the N(µ,
2) distribution, where the local variance 2 is known but the value of the local mean µ is unknown. Before the female encounters her first male, her only information about µ is that it has been randomly drawn from the 
distribution, where µ0 and are known. Initially, her best estimate is to assume that µ takes the value µ0. However, with each male she encounters, she gains information that she can use to update her estimate of µ and more generally update her overall uncertainty about the value of µ. This process of updating can be precisely quantified by modeling the process in a Bayesian framework.
Under the Bayesian model, the distribution of µ across different locations or generations can be interpreted as a subjective prior distribution, representing the female's initial beliefs about the unknown value of µ in her locality. Our assumptions imply that this distribution is 
where µ0 is now interpreted as the known prior mean of this subjective distribution and is interpreted as the prior variance (so 
is the prior precision, giving a measure of how closely her beliefs are concentrated around the mean).
Each time the female encounters another male, Bayes theorem specifies how her beliefs prior to the encounter should be updated and modified to give a posterior distribution that incorporates the information provided by the newly observed male fitness value. Consider a female who has just encountered her nth male. Denote the fitness values observed so far by x1, ..., xn, with mean 
Standard results (DeGroot 1970) show that her posterior distribution is the 
distribution, so her information state can be summarized by the posterior mean µn and the posterior variance 
(or posterior precision 
), where for n = 1, 2, 3, ...
 | (5) |
 | (6) |
decreases deterministically with each extra observation because increases by 1/2 irrespective of the observed values x1, ..., xn.
The weight ßn given to the initial estimate µ0 also decreases deterministically with each extra observation, again independently of the observed values x1, ..., xn. Note that the weight depends only on the value of n and the relative values of the residual (local) variance 2 and the environmental variance and not on their absolute values. As the local variability 2 decreases relative to 
less weight is given to µ0 and more weight is given to the data. This is to be expected because 2 decreasing means the local data are less variable and hence a more reliable indicator of µ. Indeed, if 2 was very small, then every male in a location would have roughly the same value, and the first male value observed would be a much better indicator of local male values than µ0. Alternatively, if was very small, then there would be very little variation in the value of µ seen by different females, and the problem would reduce to the setting of a constant environment with known mean µ0.
|
OPTIMAL MATE CHOICE IN A SPATIALLY VARYING ENVIRONMENT |
|---|
In a spatially varying environment, the male fitness values encountered by an individual female are independent observations from the N(µ,
2) distribution, where the value of µ varies independently from female to female (or location to location) according to the distribution.
Because the randomness affects each female in the population independently of the other population members, it is appropriate to base fitness measures on the overall net fitness gain to each individual female (Houston and McNamara 1999). Thus, we define an optimal mate choice rule in a spatially varying environment to be one that maximizes the expected net fitness gain, where this expectation is taken first over the male values experienced for each value of µ and then over the distribution of µ across locations.
Recall that an optimal action for a female at each stage is to accept the current prospective male if and only if the reward from mating is greater than or equal to the expected net future gain under an optimal rule. Consider a female who has just encountered her nth male with observed fitness value xn and is deciding whether to accept or reject this potential mate. We saw in the previous section that, having observed xn, her current state of information is characterized by the values µn and 
Define 
to be the expected future net fitness gain if she continues searching under an optimal rule, starting from the state 
Then, the optimal action at stage n is
 | (7) |
Following DeGroot (1970, p. 336–41), the optimal threshold for fixed c can be expressed as µn plus a term that depends only on while Equation 5 shows that, for fixed 2 and 
itself depends only on n. Thus, for fixed c, 2, and there is a corresponding sequence of fixed values 
(0), (1), (2), ..., such that the dependence of the optimal threshold and optimal future net fitness gain on the current information state 
has the simple form
 | (8) |
(n)depends on n but not on µ0 or x1, ..., xn and hence not on µn. DeGroot (1970) gives an iterative procedure for numerically calculating (0), (1), (2), ..., but there is no simple explicit expression for the sequence of values.
We call the rule defined by these thresholds an optimal learning rule. In particular, because the female starts the process with an initial prior distribution the overall expected net fitness gain under an optimal rule is given by
 | (9) |
In the form given by Equation 8 above, the female compares xn with a threshold in which the value of µn already incorporates the information provided by xn. From Equations 5 and 6, we can express µn in terms of xn and µn–1 and hence rewrite the inequality 
in the equivalent form 
This provides an equivalent rule for the optimal action at the nth stage of the form:
 | (10) |
Again, because changes deterministically with n, e(n) depends only on n (for given c, 2, and ), and both (n) and e(n) can be shown to converge to the same limit 
as n becomes large. Figure 2 displays the typical form of e(n) for a range of environmental variances.
|
), and Consider now how the action specified by the optimal learning rule changes in practice over a sequence of encounters. From Equation 10, the optimal action for the female at the nth encounter is to compare the new observation xn with a threshold composed of the previous Bayes estimate of the mean (the posterior mean µn–1) plus the relative threshold e(n). Before her first encounter, her estimate of the mean is µ0, and the value of the relative threshold e(1) is set somewhat high (Figure 2). The learning process enables her to take advantage of cases where the local value of µ is higher than the long-run average male fitness value µ0 and not rush to accept the first male whose value, though high, may well be exceeded in later encounters. However, as she starts to encounter further males, the weights ßn ensure that the posterior mean adjusts rapidly toward the observed mean-to-date
n–1, whereas at the same time, the relative threshold e(n) falls rapidly to the value that is optimal in a constant environment (see Equation 3). By this stage, the threshold has adjusted to a value that would be optimal in a constant environment with the same values of c and but where the local mean µ had a value equal to µn–1. This enables the female to adapt quickly in cases where the local value of µ is lower than the long-run average male fitness value µ0 and not incur unnecessary costs by continuing with a threshold that is too high.
The threshold defined by the optimal learning rule is free to rise or fall during a search, depending on the values x1, x2, x3,... experienced by the searcher. However, because the female only accepts mates of value higher than her current threshold, each time she decides to continue searching, the rejected male must have had a relatively low fitness value. This in turn feeds into her posterior mean µn, and it can be shown analytically that the expected value of the threshold used falls over time, where the expectation is over females still searching at that stage. Thus, the current posterior mean and the current threshold of a female still searching at the nth stage will also be relatively low. Figure 3 gives some indication of how thresholds fall on average as searching continues. For each n, what is plotted is the average threshold used by all females still searching at the nth stage, where the expectation is taken first with respect to the (relatively low) male values that must have been experienced to date and then with respect to the distribution for the local mean µ. Although the comment of Real (1990) that "No simple, smooth behaviour can be predicted" certainly does apply to the changing thresholds of any fixed focal female, we see that these average thresholds do fall smoothly and predictably as the number of males encountered increases.
|
Indeed, as n increases, we see from Equations 5 and 6 that more and more weight is given to the increasing amount of local data. Moreover, in the Bayesian context, the local value of µ will eventually be learned as more and more observations are taken, and the problem then reduces to the known (constant) environment model discussed above. This is exactly reflected in the behavior of the thresholds specified by the optimal learning rule; as n tends to infinity, the weight given to the observations tends to 1,
n tends to µ, and (n) tends to 
so that the optimal threshold tends to 
agreeing with the threshold Equation 3 that is optimal for the corresponding constant environment model.
Finally, we can gain some insight by comparing the optimal learning rule with more simplistic rules. One estimate of the local mean µ, based only on the long-run average value and ignoring the local observations, would be to use µ0. Another estimate, based only on the current local observations x1, ..., xn and ignoring the long-run average, would be to take the local observed average Analogy with the constant environment model suggests 2 simple but extreme possible mate choice rules: 1) accept xn if and only if xn 
µ0 + ' and 2) accept xn if and only if xn n + '', where 'and ''are appropriately chosen relative thresholds. The first rule ignores the data and compares xn with an absolute standard µ0. It might fail to take advantage of good locations or lead to a prolonged and costly search in poor locations. The second rule is a purely relative rule, comparing xn only with the female's own experience and ignoring the long-run average. It might be misled into poor decisions by unusually high or low early observations. Both rules have their drawbacks, and we can see from Equations 5 and 10 that the optimal rule depends on a weighted average of these 2 simple extreme estimates considered before, where the Bayesian analysis precisely identifies the optimal weights.
|
THE REPRODUCTIVE ADVANTAGE OF LEARNING |
|---|
In a spatially varying environment, the optimal learning rule specified by Equation 7 or 10 gives greater expected reward than any other rule. It certainly does better than any fixed threshold rule, but on the other hand, it is also much more complex than a fixed threshold rule. There may be real costs associated with, say, the increased neurological capacity required to successfully implement more complex rules. Although such costs are not explicitly modeled here, their possible presence prompts us to compare the reproductive advantage of the optimal learning rule with that of the best fixed threshold rule, so as to identify what type of conditions favor learning in a spatially varying environment.
The numerical results set out in Table 1 show that, for fixed values of the residual variance 2, the expected reward under the optimal learning rule decreases as the environmental variance increases. Conversely, for fixed values of it increases as 2 increases. The reward from using the best fixed threshold rule behaves in a similar way; it falls with for fixed 2, and it increases with 2 for fixed Moreover, the difference between the expected reward under the optimal learning rule and that under the best fixed threshold rule increases appreciably as increases. The expected reward under the optimal learning rule was calculated using Expression 9, and the other values were evaluated by repeated numerical integration. Overall, we see that the optimal learning rule has the greatest advantage when both the residual and environmental variances are relatively large.
|
Figure 4 compares the performance of the 2 types of rules in terms of the percentage increase in expected reward from using the optimal learning rule rather than the best fixed threshold rule, for a range of values of
and 2. Although the absolute values of the expected rewards increase with 2 and decrease with the percentage increases do not behave in quite the same way, as seen, for example, in the plotted values at 0 = 1. This is because the baseline (the reward from the fixed threshold rule) also increases, but at a different rate from that for the optimal learning rule. However, a similar overall picture emerges—learning is most advantageous when the environmental variance and the residual variance are both large.
|
To gain more insight into why and when learning is advantageous, it is helpful to consider what happens for extreme values of the variance parameters. When the environmental variance,
is very small, then the environment is relatively constant, and the mean is roughly the same in all locations, so (Real 1990) the optimal fixed threshold will perform well throughout. Conversely, if the environmental variance is very large, then the mean value of the prospective mates is much more variable across locations, and it is advantageous to use the observed values seen during the search to provide a much better estimate of the local mean value. If the local residual variance 2 is very small, there is very little variation in the male values around the local mean. Now a rule that accepts the first male encountered is optimal as it minimizes cost without sacrificing potential reward, and under this rule, the expected fitness of the mate accepted is equal to the local mean. Indeed, it does not matter much which male is chosen because all males in a given locality are of similar quality. Conversely, when 2 is very large, male values show high local variability; a fixed threshold rule will still do well when is small, but when is large, it may fail to take advantage of the local possibility for high rewards, and there is then strong selection pressure for rules that learn. This explains why the optimal learning rule has significantly greater advantage only when both the residual and environmental variances are large.
The complexity of the optimal learning rule leads us to investigate whether there are other learning rules that retain its reproductive advantage but are significantly less complex. With this in mind, we define the class of simple learning rules to be the set of rules of the form:
 |
to the residual variance 2. Because µ0, and 2 are all assumed to remain constant from generation to generation, the population might more easily evolve a simple learning rule that was adapted to these known values.
In our computations, the simple learning rule did nearly as well as the optimal learning rule—its performance was within 1% of that of the optimal learning rule across the full parameter range, though the difference increased very slightly as 0 increased. Thus, the numerical results and comments above apply equally to comparing the best simple learning rule with the best fixed threshold rule. This will be particularly relevant when we discuss the evolution of rules.
We now look at the thresholds themselves and, in particular, how the best fixed threshold and the fixed relative threshold of the best simple learning rule depend on the (relative) values of the environmental variance and the residual variance 2.
Recall, from Figure 1, that a fixed threshold can perform very badly if it is even slightly higher than the optimal value for its local environment. To give a good average performance across the full distribution of µ values, an inflexible fixed threshold must balance accepting lower than optimal values in cases when µ is large against incurring prohibitively high search costs when µ is small. We see from Figure 5a that the value of the best fixed threshold has the following properties: 1) it is below the long-run average male fitness value (here µ0 = 5) for a range of 0 values when is small because there is nothing to learn, 2) it increases significantly with for each fixed 0, and 3) it decreases slightly as 0 increases for each fixed , to guard against a low local value of µ, where the decrease is more pronounced for small values of than for larger values. Similarly, the value of the fixed relative threshold for the best simple learning rule increases significantly with for each fixed 0 but changes very little with 0 for each fixed (increasing, but only relatively slowly, as 0 increases).
|
The fixed relative threshold of a simple learning rule is itself a compromise between the different stage-dependent values of the optimal relative threshold e(n) (see Figure 2). In the cases considered here, e(n) is decreasing in n and quickly approaches its limiting value of
so the best value of d is always slightly above this limit. For comparison with the fixed threshold values in Figure 5a, note that a simple learning rule with relative threshold d starts off with initial acceptance threshold µ0 + d. After that, its exact value changes randomly in response to the observations. Taking µ0 = 5 here, we find that the best simple learning rule starts off initially with a higher threshold than the best fixed threshold rule, for all values of and 0. This enables it to take advantage of situations where the local value of µ is high. However, the learning component (µn–1) allows it to adjust quickly, reduce its threshold, and avoid excessive search costs in cases where the observed male fitness values indicate that the local value of µ is low. This is well illustrated in Figure 6, where we see that the average search time under the best fixed threshold rule increases dramatically in locations where µ is low. Thus, in terms of the average search time, the best fixed threshold rule does only slightly better than the optimal leaning rule in locations where µ is high but is strikingly worse than the optimal leaning rule in locations where µ is low.
|
|
EVOLUTION OF RULES |
|---|
We have analyzed optimal learning rules, but can rules of this type easily evolve? To investigate this, we simulated the evolution of learning rules using a GA. A GA can be regarded as a minimal model of the genetics rather than a realistic representation (Axelrod 1987
). In each simulation, we followed a population of size 250 for 6000 generations. Crossover and mutation were allowed, and rules were coded using a Gray code, so mutations were always to neighboring thresholds. Generally convergence had occurred within the first 1000 generations, and the results shown are averages over the last 5000 of the 6000 generations. Details can be found in Ramsey (1994), but these details are not important because results were robust to parameters such as the mutation rate and the fraction of the population replaced each generation.
We first considered a constant environment. Here we took a rule to be described by a single parameter: the acceptance threshold. In this case, the average population fitness rapidly converges to near maximum values. In particular, evolved rules easily outperformed the rule of choosing the first male encountered when the variance 2 was appreciably large. Evolved thresholds, however, fluctuated more markedly and tended to be significantly below the optimal threshold. This is consistent with Figure 1; there is low selection pressure below the optimal threshold but high selection pressure to reduce the threshold when it is above the optimal threshold.
We also considered these fixed threshold rules in a spatially varying environment. Evolved thresholds decreased as increased and increased as 2 increased, as theory predicts (see Figure 5). Thresholds were again below that under the optimal fixed threshold.
To allow learning in a spatially varying environment, we evolved rules whose form is motivated by the simple learning rules previously considered. In particular, from Equations 7 and 8, an nth male of quality xn is accepted under the optimal learning rule if and only if xn µn + (n), where, by rewriting Equation 5, µn is given by
 |
With this in mind, we specify a rule by 3 parameters m0, d, and r, where 0 < r < 1. Under a rule with these parameters, the nth male with quality xn is accepted if and only if
 |
 |
, reflecting the relative weight given to the prior mean m0 and the observed mean 
and thus the speed of learning. We considered values of 2 and lying in the range {0.25, 0.5, 1, 2, 4}. We evolved rules repeatedly for each of the 25 combinations of these parameters. When such rules evolve, there is initially strong selection pressure against high values of d. Convergence is slower than for fixed threshold rules, but there was convergence by generation 1000.
As noted above, the theory predicts that when 2 is small, it does not matter much which male is chosen because they are all of similar quality, and the selection pressure is low provided achieved thresholds are sufficiently low. For the evolution of rules, this neutrality means that there may be significant drift in evolved parameters, and it may be difficult for the more complex rules to home in on exactly the right combinations of rule parameters. Taken together with the continuing effects of mutation and crossover, this means there may still be noticeable variation and possibly underperformance within a population using complex rules, even when it is close to convergence. When 2 is large, the theory predicts that the performance of the best fixed threshold rule will be comparable to that of the optimal learning rule when is small but may be significantly worse than the optimal learning rule when is large as there is then strong selection pressure for rules that learn. Figure 7 clearly illustrates these points for 2 large, with low selection pressure leading to evolved fixed threshold rules actually doing slightly better than evolved simple learning rules when is small but doing significantly worse as increases.
|
In all cases, regardless of the strength of selection, there was high variability in parameters and correlations in parameters across successive runs with the same parameter values. When
is low, we see from Figure 8a that m0 and d are highly negatively correlated. This correlation can be understood as follows. In this case, the environment is approximately constant. Thus, learning is not valuable, and fixed threshold rules perform well. A learning rule with r = 1 is just a fixed threshold rule with threshold m0 + d. This threshold can be achieved by different combinations of the 2 parameters m0 and d. In particular, a range of values of m0 can achieve the optimal threshold provided that as m0 is increased d is decreased to compensate. Of course, the behavior observed under a rule depends on how the threshold changes with experience. Thus, high variability in m0 and d does not necessarily imply high variability in what would be observed.
|
When the environmental variance
is large, Figure 8b shows that there is a high negative correlation between the values of r and d used by members of the evolved population. The intuition is that the female needs to be very responsive to the locally observed values because the mean of the distribution varies so much from location to location. For example, if µ is low and the relative threshold d is high, then the female may take a very long time to accept a male; she will incur considerable search costs unless she quickly lowers her threshold in response to encounters with poor-quality males, and this responsiveness corresponds a low value of r. However, if d is lower, the female does not need to be so responsive. Thus, there are a variety of different ways of achieving near-optimal behavior, ranging from adapting fast with a high relative threshold to adapting more slowly but with a lower relative threshold, and a good simple learning rule must use a low r value if its d value is high and vice versa. Our results indicate that different populations could have found different ways of solving the same problem, for example, using different combinations of values of r and d, and these different r and d values will result in observable differences in behavior. A corollary of this is that observed differences in female behavior between populations does not necessarily imply that females are facing different problems or even using different types of rule.
|
TEMPORALLY VARYING ENVIRONMENTS |
|---|
When fluctuations are temporal as opposed to spatial, all females in the population experience the same value of µ in a given year. Thus, if µ varies (i.e.,
> 0), all females are subject to the same environmental fluctuations, and natural section maximizes geometric mean fitness (Lewontin and Cohen 1969). To define the fitness of a particular rule, let V(µ) be the expected payoff to a female using the rule when the environmental mean is µ. Then, the geometric mean fitness of the rule is G = exp(g), where
 | (11) |
Because of the logarithm in Expression 11, geometric mean fitness is highly sensitive to low values of V(µ). This means that it is more important to avoid poor performance for some values of µ than to achieve good performance for other values. We can therefore expect the optimal learning rule to be conservative, avoiding risks so as to perform reasonably in all years.
In this situation, it is impractical to find optimal learning rules with this fitness criterion. Instead, we evolved rules using the GA. When fixed threshold rules are evolved, we see from Figure 9 that the resulting thresholds are significantly lower than in a spatially varying environment with the same parameters, especially for high This is as expected because low thresholds reduce the risk of a very low payoff in bad years (low µ). However, when simple learning rules are evolved, there is surprisingly little difference to the spatially varying case. The exception is that r is significantly lower. This is so that in a bad year the threshold is lowered quickly in response to encounters with poor-quality males, so reducing the risk of a long and costly search.
|
|
DISCUSSION AND COMPARISON WITH PREVIOUS MODELS |
|---|
We introduce a model for optimal mate choice in situations where the observed values of prospective mates are subject to 2 types of variations—variation of individual observations about the (local) mean and variation of the local mean about some long-run average value.
Each female observes prospective mates in a single location, rather than the multiple patch model of Hutchinson and Halupka (2004). In contrast to the "full knowledge" models of Real (1990) and Janetos (1980), we assume that the local mean is unknown and can vary from location to location or season to season. Under our model, the local male population is possibly infinite and is observed sequentially with no possibility of recall, so the best-of-n rule (Janetos 1980) is not relevant, and finite population models (Dombrovsky and Perrin 1994) are excluded. The value of each male encountered is observed exactly, obviating the need for reinspection or a comparative Bayes approach (Luttbeg 1996, 2002; Fawcett and Johnstone 2003b). We emphasize an optimality framework with a single choosy sex, rather than a game theoretic framework (McNamara and Collins 1990; Johnstone 1997; Bergstrom and Real 2000; Fawcett and Johnstone 2003a), and our optimality criterion is based on a single measure combining both the value of the mate chosen and the search costs, as opposed to simply maximizing the value of the mate (Mazalov et al. 1996) or maximizing the probability of choosing the best mate (Dombrovsky and Perrin 1994).
In a constant environment, where a fixed threshold rule is known to be optimal (Real 1990), we show that there will be strong selective pressure against females that use thresholds higher than this optimal value, but there may be only weak selective pressure against females that use lower than optimal thresholds. In a varying environment, the long-run performance of a mate choice rule must be evaluated in terms of its average performance over the whole range of environmental conditions and not just by its performance in a single fixed environment. We show that the optimal rule is a learning rule, under which the acceptance threshold is composed of the current estimate of the mean plus a relative threshold that depends only on the number of males encountered to date, and demonstrate how learning enables the female to take advantage of the local information through the changing weight given to the local mean. We compare the performance of the optimal learning rule with both fixed threshold rules and simpler learning rules, analyzing the effect of different levels of variability on the performance of these rules. We show that the relative advantage of learning is greatest when both residual and environmental variances are large because this results in high local variability about a very variable mean and therefore strong selection pressure for learning. Although the acceptance threshold of an individual focal female may vary unpredictably over her search (Real 1990), we show that under the optimal learning rule, the expected threshold used falls smoothly as a function of the number of males encountered, when averaged over females still searching at that stage. Results for temporally varying environments are not presented in detail as they are qualitatively similar to those for spatially varying environments. However, they do indicate even greater selective pressure against setting thresholds too high.
We also break new ground in quantitatively and qualitatively exploring the evolution of these different rules, again looking at the effect of different levels of variability. The simulated evolutions indicate that evolved fixed thresholds will be below the optimal fixed threshold in both constant and spatially varying environments, reflecting the selective pressure against erring on the high side. In spatially varying environments, the evolved thresholds decrease as the environmental variance increases and increase as the residual variance increases. For simple learning rules, our results indicate high variability in the evolved rule parameters for different members of the population, emphasizing the variety of ways in which a near-optimal rule can be expressed.
Overall, Mazalov et al. (1996) comes perhaps closest to our analysis in spirit and content. They consider a finite horizon sequential learning model when local male values have a normal distribution with unknown mean (and possibly variance). They derive an optimal learning rule, in some ways similar to ours in structure, based on updating estimates of the local mean and variance and compare it with the fully adapted rule that would be optimal if the parameters were known. However, despite the common elements, our treatment differs from theirs in several crucial respects.
Firstly, their model has no search costs, so their criterion for an optimal rule is just that it maximizes the expected value of the male obtained, and expected search times do not influence the choice of an optimal rule. Our optimality criterion combines both rewards and costs into a complete overall measure of a rule's performance. This leads us to very different conclusions about the properties of an optimal rule and the relative performance of fixed and learning rules in a varying environment.
To illustrate the effect of the difference in optimality criterion, consider the following oversimplified analysis of how a fixed rule fully adapted to a given local mean would perform relative to an optimal learning rule in an environment with a different local mean when the horizon is large. For the example computed in Mazalov et al. (1996), very roughly speaking, the fixed rule has a similar performance to their learning rule when the actual local mean is much lower than the value to which the rule is adapted (essentially because it may eventually accept a relatively high-quality male, though with small probability on each encounter) but performs significantly worse than the learning rule when the local mean is higher than the value to which the rule is adapted (now it may accept a relatively low-quality male). In a similar situation, under our model and optimality criterion, the discussion earlier in this paper indicates just the opposite—the fixed rule has a much worse performance than our learning rule when the actual local mean is much lower than the value the rule is adapted to (it may be penalized for the long search time resulting from an over-high threshold) but has a similar performance to the optimal learning rule when the local mean is higher than the adapted value (its low search cost may compensate for a low value of the male accepted).
A second crucial difference is that Mazalov et al. (1996) assume that females have no prior knowledge of the parameters of the local distribution of male quality, and parameter estimation is based purely on the observed values. This has the particular disadvantage that under their model, the female is not allowed to accept the first male encountered—however high their value—but is constrained to use this value as an initial estimate of the local mean against which to judge the next encounter and update as appropriate. We give a fully Bayesian treatment of inference for the local mean, assuming that females have adapted to the relevant long-run parameters.
Finally, we note that our comparisons of different types of rules are made under the assumption that recall is not allowed. The advantage of learning may be even greater if recall of previously rejected prospective mates is possible. In a constant environment, µ is effectively known, and the observations are independent given µ, so they carry no extra information (about µ), and a learning rule can do no better than a fixed threshold rule. In a varying environment, the observations do carry information about µ. A learning rule can take advantage of this information to lower its threshold in response to low observed male fitness values. In such cases, a prospective mate rejected earlier may later seem much more attractive. However, a fixed threshold rule can never take advantage of this information; a prospective mate that was not acceptable at one point in time would always have a value below the fixed threshold and would never be acceptable despite any indications that the local mean value was unexpectedly low. In particular, a learning rule will be favored over a fixed threshold rule in cases where recall and search costs are very low and recall is reliable, such as a female bird choosing a mate from a male lek.
|
ACKNOWLEDGEMENTS |
|---|
We thank Alasdair Houston, Barney Luttbeg, and Peter Todd for their comments on earlier drafts of this paper. John McNamara acknowledges the support of the Leverhulme Trust. We also thank the editor and 2 anonymous referees for their helpful and perceptive comments.
|
REFERENCES |
|---|
Alatalo RV, Carlson A, Lundberg A. 1988. The search cost in mate choice of the pied flycatcher. Anim Behav 36:289–91.[CrossRef][Web of Science]
Axelrod R. 1987. The evolution of strategies in the iterated prisoner's dilemma. In: Davis L, editor. Genetic algorithms and simulated annealing. Los Altos, CA: Morgan Kaufmann publishers Inc. p 32–41.
Bakker TCM, Milinski M. 1991. Sequential mate choice and the previous male effect in sticklebacks. Behav Ecol Sociobiol 29:205–10.[CrossRef][Web of Science]
Bergstrom CT, Real LA. 2000. Towards a theory of mutual mate choice: lessons from two-sided matching. Evol Ecol Res 2:493–508.
Collins EJ, McNamara JM. 1993. The job-search problem with competition: an evolutionarily stable dynamic strategy. Adv Appl Probab 25:314–33.[CrossRef]
Collins SA. 1995. The effect of recent experience on female choice in zebra finches. Anim Behav 49:479–86.[CrossRef][Web of Science]
DeGroot MH. 1970. Optimal statistical decisions. New York: McGraw-Hill.
Dombrovsky Y, Perrin N. 1994. On adaptive search and optimal stopping in sequential mate choice. Am Nat 144:355–61.[CrossRef][Web of Science]
Downhower JF, Lank DB. 1994. Effect of previous experience on mate choice by female mottled sculpins. Anim Behav 47:369–72.[CrossRef]
Fawcett TW, Johnstone RA. 2003a. Mate choice in the face of costly competition. Behav Ecol 14:771–9.
Fawcett TW, Johnstone RA. 2003b. Optimal assessment of multiple cues. Proc R Soc Lond B Biol Sci 270:1637–43.[Medline]
Houston AI, McNamara JM. 1999. Models of adaptive behaviour. Cambridge, UK: Cambridge University Press.
Hutchinson JMC, Halupka K. 2004. Mate choice when males are in patches: optimal strategies and good rules of thumb. J Theor Biol 231:129–51.
Janetos AC. 1980. Strategies of female mate choice: a theoretical analysis. Behav Ecol Sociobiol 7:107–12.[Medline]
Johnstone RA. 1997. The tactics of mutual mate choice and competitive search. Behav Ecol Sociobiol 40:51–9.
Lewontin RC, Cohen D. 1969. On population growth in a randomly varying environment. Proc Natl Acad Sci USA 62:1056–60.
Luttbeg B. 1996. A comparative Bayes tactic for mate assessment and choice. Behav Ecol 7:451–60.
Luttbeg B. 2002. Assessing the robustness and optimality of alternative decision rules with varying assumptions. Anim Behav 63:805–14.[CrossRef][Web of Science]
Mazalov V, Perrin N, Dombrovsky Y. 1996. Adaptive search and information updating in sequential mate choice. Am Nat 148:123–37.[CrossRef][Web of Science]
McNamara JM, Collins EJ. 1990. The job-search problem as an employer-candidate game. J Appl Probab 28:815–27.[CrossRef]
Milinski M, Bakker TCM. 1992. Costs influence sequential mate choice in sticklebacks, Gasterosteus aculeatus. Proc R Soc Lond B Biol Sci 250:229–33.
Parker GA. 1983. Mate quality and mating decisions. In: Bateson P, editor. Mate choice. Cambridge, UK: Cambridge University Press. p 141–66.
Ramsey DM. 1994. Models of evolution, interaction and learning in sequential decision processes [dissertation]. Bristol, UK: Department of Mathematics, University of Bristol.
Real L. 1990. Search theory and mate choice. I. Models of single-sex discrimination. Am Nat 136:376–404.[CrossRef][Web of Science]
Simao J, Todd PM. 2002. Modelling mate choice in monogamous mating systems with courtship. Adapt Behav 10:113–36.[CrossRef]
Svensson EI, Sinervo B. 2004. Spatial scale and temporal component of selection in side-blotched lizards. Am Nat 163:726–34.[CrossRef][Web of Science][Medline]
Wong BBM, Candolin U. 2005. How is female mate choice affected by male competition. Biol Rev Camb Philos Soc 80:559–71.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  T. W. Fawcett and C. Bleay Previous experiences shape adaptive mate preferences Behav. Ecol., January 1, 2009; 20(1): 68 - 78. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||