[HR][/HR]Ilways set up one hive in my yard .with painted with bright colors. I then use water mellon as a rate. Sometime I will bab some me 20 20 on a cotton ball as a lure. If they are attracted to that home I know that I have a problem.
DETERMINING LOW LEVELS OF AFRICANIZATION
IN UNMANAGED HONEY BEE COLONIES
USING THREE DIAGNOSTIC TECHNIQUES
A thesis submitted to the Faculty of the University of Delaware in partial
fulfillment of the requirements for the degree of Master of Science in Entomology
Copyright 2013 Katherine Darger
All Rights Reserved
DETERMINING LOW LEVELS OF AFRICANIZATION
IN UNMANAGED HONEY BEE COLONIES
USING THREE DIAGNOSTIC TECHNIQUES
Katherine Eva Darger
Deborah A. Delaney, Ph.D.
Professor in charge of thesis on behalf of the Advisory Committee
Douglas Tallamy, Ph.D.
Chair of the Department of Entomology and Wildlife Ecology
Mark W. Rieger, Ph.D.
Dean of the College of Agriculture and Natural Resources
James G. Richards, Ph.D.
Vice Provost for Graduate and Professional Education
I would like to acknowledge the citizen scientists and enthusiastic beekeepers
that located and collected honey bees from non-managed nests from as far as Illinois
and Maine. I would like to thank Dr. David R. Tarpy for sharing his molecular
biology knowledge and advice, Dr. David De Jong for lending his morphometric
expertise, and Dr. Warren Booth for his assistance developing microsatellites. Dr.
Bruce Kingham at the Delaware Biotechnology Institute sequenced microsatellite
samples. Dianna Sammataro and Jerry Hayes, both of the USDA during the time of
the project, provided samples of known Africanized honey bees. Dr. Brenna Traver of
Virginia Tech sent samples of Africanized bees from Belize. Dr. W. S. Sheppard
provided honey bees from Africa. Furthermore, Cindy Hodges of the Metro Atlanta
Beekeepers provided priceless insight by navigating and hosting me during my
collection trip to Georgia. Carolyn Thomas assisted with laboratory procedures and
cut numerous bee legs. Angela Carcione re-extracted DNA and Judith Keller lent her
molecular expertise to refute questionable diagnoses. Mostly, I’d like to acknowledge
the endless help from my committee members: Dr. Charles Bartlett, Dr. Karl Schmidt,
Dr. Blake Meyers, and especially my advisor, Dr. Deborah Delaney.
TABLE OF CONTENTS
LIST OF TABLES ......................................................................................................... v
LIST OF FIGURES ....................................................................................................... vi
ABSTRACT ................................................................................................................. vii
Chapter 1 INTRODUCTION: .......................................................................................... 1
Significance of Apis mellifera ............................................................................ 1
Suspect activity/signs of Africanization ............................................................. 8
Molecular Identification ................................................................................... 11
mtDNA ....................................................................................................... 11
Microsatellite Analysis ..................................................................................... 12
Hypothesis ........................................................................................................ 13
Methods and Materials: .................................................................................... 13
Collection Technique .................................................................................. 13
Total DNA Extraction ................................................................................ 14
Microsatellites ............................................................................................ 16
Results .............................................................................................................. 18
Morphometrics ........................................................................................... 18
Mitochondrial Testing ................................................................................ 20
Microsatellites ............................................................................................ 22
Discussion ................................................................................................... 23
REFERENCES ............................................................................................................. 31
Appendix ..................................................................................................................... 36
PERTINENT GRAPHS AND FIGURES ........................................................ 36
LIST OF TABLES
Table 1 Mitotypes exhibited in Africanized Populations. .................................... 36
Table 2 The average number of alleles (na), allelic richness (ag), and expected
heterozygosity (He) for each population. ................................................. 37
Table 3 Pairwise FST values for managed, unmanaged, Africanized bees from
Florida, Africanized bees from Arizona, Africanized bees from Belize,
and fully African bees from Africa and Brazil. ....................................... 37
Table 4 P-values for genotypic (above diagonal) and genic (below diagonal)
differentiation for 6 populations based on microsatellite data. ............... 38
Table 5 Number of samples collected from each state. Due to the low
numbers from certain states they were combined into regions during
the population structure evaluation, such as the bees from Africa
combined with those fully Africanized from Brazil. ............................... 38
Table 6 Allele frequency data for 13 microsatellite loci used in analyzing the
six populations. The highlighted numbers show an indication that this
allele at this allelic concentration may be indicative of Africanization.
Those highlighted in blue show high levels of allelic concentration in
every Africanized population. ................................................................. 39
Table 7 Percentage of mitotypes in unmanaged and managed populations by
state. ......................................................................................................... 44
Table 8 These are the proportions of membership to each of the two inferred
clusters for each pre-determined population, corresponding with the
STRUCTURE bar figure results above (Figure # 1). .............................. 46
LIST OF FIGURES
Figure 1 A partial map of the subspecies of Apis mellifera, not including the
Asia or the borders of the region of the species Apis cerana. ................... 3
Figure 2 All samples of unmanaged honey bees clustered together, separate
from the stock Apis mellifera carnica, A. m. caucasica, A. m. ligustica,
or A. m. mellifera. Image courtesy of Dr. David De Jong and his lab
at the University of Sao Paulo, Brazil. .................................................. 19
Figure 3 Figure 3: All symbols to the left of zero are Africanized using the
Geometric Wing Venation (GWV) test while those to the right do not
measure as Africanized using GWV. All samples in this figure tested
positive for Africanization using the USDA-ID test. Figure courtesy
of Dr. David De Jong. .............................................................................. 20
Figure 4 Figure 4: Bar plots of populations as determined by the program
STRUCTURE. Each comparison of sample clusters sorted into two
populations, as represented by the two colors. To further explain,
under bar “A” the unmanaged, managed, samples from Washington,
and Africanized samples shared the most common alleles with each
other. They did share some with the second population presented,
explaining the slight blend of colors between the two populations.
Note the uniform separation of color among all samples represented in
bar “G”, showing that this is one population. .......................................... 45
“Killer bees” arrived in the United States in the year 1990. Questions have
arisen regarding low levels of Africanization in regions bordering the locations with
established, Africanized bees. Honey bees were collected and examined using three
methods of testing to determine levels of Africanization. With morphometrics,
mitochondrial DNA, and nuclear DNA tested with the use of microsatellites we found
that the known Africanized bees collected by the Florida Department of Agriculture
did not exhibit Africanization other than in the preliminary, morphometric test
performed by the Department of Agriculture.
Significance of Apis mellifera
Insect pollinators are responsible for one third of the human diet. Pollination
services by honey bees have a value of $15.12 billion in the United States in 2009
(Calderone 2012). The honey bee is the most prolific pollinator based on efficiency,
the wide range of targeted plants, and lack of damage to the plant (McGregor 1976).
Honey is also ingrained in modern culture. Honey can be found in cereal, coffee
sweetener, bread, lip balm, soap, and hair conditioner. In Canada, the value of honey
produced in 2009 was worth $125.3 million (Statistics Canada 2010). Because honey
bees serve a crucial role, many beekeepers attempt to maintain a strong colony by
replacing the queen every year, which supports a large commercial queen breeding
industry in the United States.
Ecologically and economically, honeybees provide crucial services and so it is
essential to be able to manage them. Understanding the true introgression of
Africanized bees into the landscape of European bees is imperative as their spread
may affect beekeepers’ management practices and purchases. Many queen breeders
operate in the southern states due to the climactic allowance for brood production at an
earlier time than in the northern states. If there is a possibility of Africanized bees,
queen breeders may have to change their practices due to the potential for mating with
Africanized drones or usurpation by Africanized swarms. The public will also want to
know the extent of the spread of genes, since Africanized bees are notorious for
aggressive behavior. Scientifically, testing the maternity and paternity of honey bee
samples is an interesting method of discerning how an invasive population affects and
invades the established occupants across a region.
To address these matters, this study utilizes and compares different diagnostic
methods to develop and identify intermediate to low levels of Africanization in
unmanaged honey bee samples collected from Maine to Florida. By using nuclear
markers in conjunction with standard morphometric and mitochondrial analyses it is
possible to identify the true extent of maternal and paternal contribution of
Africanized genes into these unmanaged populations.
Honey bee subspecies were first demarcated with morphometric techniques, as
can be viewed in the following map (Figure 1).
Figure 1 A partial map of the subspecies of Apis mellifera, not including the Asia
or the borders of the region of the species Apis cerana.
At least twenty-four different subspecies, or races, can be discriminated within
the species Apis mellifera L. (Ruttner 1988; Sheppard 1997; Sheppard and Meixner
2003) spanning a vast endemic range through seasonal evergreen forests and
mountains of Europe, the savannahs and tropical regions of Africa (Figure 1), and the
montane grasslands and deserts of central and western Asia. Due to the varying
climates and ecologies present in this native range differences in behavior and
morphology have evolved (Rinderer 1986).
Subspecific groupings were based on morphological and behavioral attributes,
which are now upheld by DNA testing. Honey bee subspecies of tropical Africa,
including A. m. scutellata, have high swarming rates due to rapid growth of colonies in
response to climate variations, the propensity to abscond and migrate, and the use of
unsheltered nesting sites (Wilson 1988). This behavior developed in response to
limited food resources and tropical temperatures within Africa. Swarming is the
natural method of colony population maintenance and division as well as a way to
disperse through the landscape. The high rate of swarming focuses the majority of
energy and resources on brood rearing rather than honey production, making this
subspecies unfavorable to the commercial honey bee industry and unlikely to
overwinter (Wilson 1988).
Examples of Mediterranean and southeastern European subspecies are A. m.
ligustica and A. m. carnica (Ruttner 1988). Apis m. ligustica‘s original distribution is
Italy, and A. m. carnica developed in the Southern Austrian Alps through Hungary,
Romania, and Bulgaria (Sheppard 1989). Due to the ecological pressures
characteristic of regions with polar seasons, these subspecies acclimate to periods of
warm temperatures as well as harsh winters. The honey bees of the central
Mediterranean and southeastern Europe are known for their overwintering abilities, in
which the brood cycle is interrupted to focus energy on thermoregulation. They also
have a low tendency to swarm. Descendants of these two subspecies are the most
common honey bee strains found in the United States, where the ability to overwinter
is a necessity.
Honey bees are not native to the United States; rather they were introduced by
European settlers. Initial importation attempts were unsuccessful and the hives died
on the ships. In the 1600’s, settlers successfully delivered a limited number of honey
bee colonies across the ocean. The first subspecies reported to make the over-sea
journey was A. m. mellifera. There are no known records indicating the arrival or
importation of any additional subspecies of Apis mellifera until the mid-1800’s
(Sheppard 1989). Apis m. mellifera had greater than two centuries to proliferate and
establish a feral honey bee population in the forested regions of eastern North
America. In the period between 1859 and 1922, there was an increase of importation
as beekeepers experimented with different subspecies. Seven additional subspecies
were brought into North America and tried as potential breeding stock for a
developing beekeeping industry (Sheppard 1989). The US Honeybee Act of 1922
halted further importation in response to honey bee losses on the Isle of Wight. This
mysterious bee die off was linked to the identification of the honey bee parasite,
Acarapis woodi (Rennie) in Europe (Needham et al. 1988). The importation of adult
honey bees into the United States is still prohibited; however, honey bee germplasm of
various subspecies is currently being brought in under supervision by the Animal and
Plant Health Inspection Service (APHIS). The unique importation history of Apis
mellifera into North America has led to the creation of two genetically differentiated
honey bee populations: the feral honey bee population, composed of a higher
percentage of bees representative of A. m. mellifera, and the commercial or managed
honey bee population, largely controlled by queen breeders and composed of bees
representative of A. m. ligustica and A. m. carnica (Sheppard 1989).
Apis mellifera scutellata, the African subspecies introduced into Brazil whose
hybrid is known in the Americas as the Africanized bee, is endemic to East and South
Africa, ranging from woodlands at high altitudes to tropical regions. The climactic
range includes a dry and short rainy season, punctuated by frequent brush fires. Apis
m. scutellata tend to nest in small nest cavities, including trees, holes in the ground, or
rock ledges. When a bloom occurs that offers high nectar flow, the colonies will
frequently swarm due to the restricted space of their nest cavities. Apis m. scutellata
will also swarm if there’s a low nectar supply or abscond to regions with better
resources (Rinderer 1986). The prolific foraging ability was coveted by beekeepers
of South America, where the honey bees originating from Europe were not
The introduction of A. m. scutellata into South America in the early 1950’s has
permanently affected the genetic landscape of honey bee populations in the Americas.
Their importation began as a breeding project to increase honey production of
European bees in the tropical climate of the Amazon Basin. The idea was to create a
“super bee” by producing a hybrid between European and African subspecies, a bee
with good temperament and proficient foraging behavior (Winston 1992). Swarms of
A. m. scutellata escaped before the breeding project was completed in Brazil in 1957,
and unfortunately many of the defensive Africanized honey bees quickly moved north
from Brazil through Central America to Texas, Arizona, and Florida at an
unprecedented rate of 300 km per year (Rinderer 1986). As Africanized honey bees
spread into new regions genetic introgression was first observed in unmanaged
populations. Unmanaged honey bees differ significantly from managed hives because
of the lack of human influence on genetics through re-queening (Schiff et al. 1994,
Delaney et al. 2009).
One method of range expansion the Africanized honey bee exhibited was
expansion through usurpation. Africanized honey bees abscond more frequently in
smaller clusters and will usurp established European honey bee colonies (Vergara et
al. 1989). The swarm will cluster under a hive for a short period of time, then move
in and take over (Vergara et al. 1993). This behavior is idiosyncratic to Africanized
honey bees. According to Vergara et al. (1989), “raiding swarms are most prevalent
during the early stages of the Africanization process.” There is a statistically
significant difference between usurpation of queenless colonies compared to strong
colonies, which are characterized by having large population numbers as well as
young, strong queens with a high production of queen-produced pheromones, thus
implying that a usurping cluster under the hive evaluates the level of reproductive
potential via hormone detection and targets weaker colonies (Schneider 2004).
African queens emerge earlier than European queens, allowing time to destroy the
competitive queen cells. Africanized honey bees with African patrilines also convey a
A. competitive advantage over European queens, showing greater aggression via
increased vibration, and successful elimination of rival queens (Schneider &
Suspect activity/signs of Africanization
In addition to usurpation of European honey bee colonies, Africanized honey
bees have been identified based on characteristic behavior. Africanized colonies have
been recorded hanging from limbs on trees and also colonizing in the ground.
Africanized honey bees have a more sensitive defensive response and will react to
threats near the hive faster, sting in greater quantities, and will travel longer distances
to dispel a threat. This aggressive defense response has been the source of their
infamy. Anecdotal evidence and observations of novel honey bee behavior have
caused the beekeepers and industry to question the levels of Africanization in the
unmanaged bees establishing nests along the East Coast.
Previously, African introgression has been tested using mitochondrial and
morphometric analyses. The accuracy of these diagnostic tools is limited when used
on populations with low or intermediate levels of Africanization or Africanized
populations occurring at the edge of their optimal range. Also, these tests do not
discriminate paternal genetic contributions. Through the use of a microsatellite kit,
population structure and interaction can be inferred. This study explores the
Africanization process threefold, by studying morphometrics, mitochondrial DNA,
and comparing them to microsatellites.
Morphometric methods of diagnosis are based on physical and structural
variations that are universally distinctive to a population, genus, species, or
subspecies. These techniques are less expensive than other laboratory techniques and
require fewer laboratory supplies; however they are time consuming and labor
Morphometric classification of honey bees has evolved over the past century.
Among the earliest morphometric classification techniques were those of Cochlov in
1916. He compared the length of the “tongue,” more accurately called the
“proboscis.” Some of Goetze’s (as cited in Ruttner 1988) comparisons included hair
length on the tergites, width of the first tarsal segment, and longitudinal diameter of
tergites 3 and 4. Ruttner’s (1988) descriptive technique became the established
method later in the century (Ruttner 1988). Rather than relying mainly on color, as
some predecessors did (e.g. Alpatov 1929), Ruttner based his classifications on
position and measurement of different parts of the head, abdomen, legs, fore wing, and
hind wing. For example, the head would be characterized by the relation of the width
of the head as compared to the width of the thorax, position of the ocelli, and length of
the proboscis and antennae (Ruttner 1988). Modern morphometric methods for
identifying Apis mellifera subspecies include four techniques: Fast Africanized Bee
Identification System (FABIS), Universal System for Detecting Africanization
Identification (USDA-ID), Automatic Bee Identification System (ABIS), and
Geometric Wing Venation.
Fast Africanized Bee Identification System (FABIS) is the field test for
preliminary identification of suspected Africanized honey bees (Rinderer 1986).
FABIS measures forewing length, fresh weight, dry weight, and femur length. This
process takes approximately twenty minutes per colony (Sylvester and Rinderer 2009).
Once the colony is identified with FABIS, the sample is analyzed with USDA-ID.
Universal System for Detecting Africanization (USDA-ID) (Sanford 2006) is
the test necessary to officially declare a case of Africanization in the Unites States.
This process is laborious due to the obligation to accuracy. Twenty-five mounted
parts of each specimen are needed to determine whether a colony is Africanized. This
requires thorough training to insure accuracy, as it includes miniscule measurements,
like the number of hamuli on the hindwing, the length and width of the basitarsus, and
the distance between wax mirrors (Rinderer et al. 1993).
Two newly established morphometric methods of analysis are less expensive
and faster than the well-established methods. The Automatic Bee Identification
System (ABIS) uses a comparison of plotted wing-vein junctions with a digital image
of the forewing of the specimen. The process takes two minutes per sample and the
accuracy rate is estimated to be 98.05% among bee species and 94% among honey bee
subspecies (Francoy et al. 2008). Geometric morphometric analysis is used to
delineate between honey bee subspecies. The test takes five minutes (Francoy et al.
2006) and has a 99.2% estimated accuracy rate looking at the wing venation angles
(Francoy et al. 2008, Francoy et al. 2009). Due to its precision and short preparation
time, this test was used on the samples in this study.
Mitochondrial DNA (mtDNA) testing is an established method of
discriminating subspecies within the species Apis. However, mtDNA is passed
maternally, and therefore, only reveals the genetic identity of the queen, so while
providing answers of the maternal inheritance, it does not account for paternal
inheritance. Upon onset of Africanized bees relocating into a particular area, the
mtDNA analysis is able to detect their presence if it is queen-driven (Hall &
Muralidharan 1989). There are different methods for determining the haplotype or
mitotype of a honey bee (Nielsen et al. 1999). The method chosen for this project
requires use of Polymerase Chain reaction (PCR) amplification of the COI-COII
intergenic spacer region within the mtDNA. This region is polymorphic and
differentiates between evolutionary lineages; African (A), western European (M), and
southeastern European (C) (Garnery et al. 1992, Franck et al. 2001, Francoy et al.
2009, Delaney et al. 2009). Digestion of the COI-COII intergenic spacer region with
the restriction enzyme Dra I, provided restriction fragment length polymorphisms
(RFLP) of over 19 mitotypes and differentiated within subspecies, specifically more
than 5 African mitotypes (Garnery et al. 1993, Franck et al. 2001). The EcoRI enzyme
has been used in the past but it has been reported that it does not distinguish certain
samples of the M lineage from the A lineage (Clarke et al. 2001). Due to the
inaccuracies of this restriction enzyme in differentiating between these lineages, the
EcoRI RFLPs were not used in determining mitotype or haplotype for this study.
Microsatellites were used in addition to RFLPs due to the extra information
they impart. Microsatellites are genomic, intergenic regions which consist of
repeating nucleotides, allowing for observations of inheritance outside of the maternal
inheritance of mtDNA. This analysis provides the ability to observe the gradients of
genetic inheritance from Africanized bees, showing both the maternal (queen) and
paternal (drone) inheritance of the resulting offspring. . Previously, microsatellites
have been used to show population dynamics over time, such as Clarke et al. (2002),
who demonstrated that the bees of the Yucatan Peninsula had marked changes over
time due to the invasion of A. m. scutellata. Through the comparison of
microsatellites, Pinto et al. (2005) showed that the interacting populations in southern
Texas changed from that of solely European genetics to a blend of A. m. scutellata
with European influences. Estoup et al. (1995), used microsatellites to support the
idea of three lineages previously upheld by mitochondrial DNA and morphometrics.
Another study conducted in Central Mexico by Kraus et al. (2007) compared mtDNA
(RFLPs) to nuclear DNA (microsatellites) and found that DNA introgressed into
populations at uneven levels. As suggested by the findings in Kraus et al. (2007), the
C mitotypes are replaced more readily by A mitotypes than M mitotypes.
I hypothesize that unmanaged honey bee colonies have blends of genes from
European and African influences and low levels of Africanization occur throughout
the United States as shown with morphometrics and DNA testing. Through the use of
the University of Delaware Entomology and Wildlife Ecology Molecular Lab, we
aimed to develop diagnostic tools that are sensitive to low-level hybridization,
determine true levels of genetic introgression, identify the geographic spread of
Africanization across a clinal gradient, and predict the advancement of Africanization
into new regions.
Methods and Materials
143 samples were collected from unmanaged honey bee colonies from South
Carolina to Connecticut during the spring and summer in 2008-2009. During the
spring and summer of 2011, I collected 72 samples and added to the unmanaged pool
(Table 5). Unmanaged colonies are often found in abandoned structures and hollow
trees and some have been known in the community for over fifteen years. Each
sample represents a colony and consists of fifteen to eighty workers. Samples were
stored in ethanol at ambient temperatures in the Molecular Entomology Lab found in
Townsend Hall of the University of Delaware until 2012 when they were transferred
to -80°C storage to lengthen the potential preservation. Known Africanized samples
from Arizona and Florida were supplied by Diana Sammataro and Jerry Hayes at the
USDA-ARS Carl Hayden Honey Bee Research Center and the Florida Department of Agriculture and Consumer Services; Bureau of Plant and Apiary Inspection,
respectively. The samples from the Florida Department of Agriculture were collected in Alabama, Georgia, and Florida but were grouped together for this study. The samples from Jerry Hayes, formerly of the Florida Department of Agriculture, were grouped together to increase the samples size. Known Africanized samples from South America as well as African samples from Pretoria and Kenya were provided by Dr. Walter S. Sheppard’s personal collection from Washington State University. Forty known Africanized honey bee samples were sent from Belize by Brenna Traver. Altogether, there are 82 samples of known African/Africanized honey bees.
For our morphometric diagnoses, ten right forewings per colony from an
assortment of colonies among our sampling collection were removed (Table 5). Our
samples and the control samples were diagnosed in Brazil by Dr. David De Jong using
the geometric morphometric technique of comparing wing venation (Francoy et al.
2006, Francoy et al. 2008, Francoy et al. 2009). The results are displayed in Principle
Component Analysis figures to exhibit the clustering of populations according to
Total DNA Extraction
A hind leg from two bees per sample, totaling in 188 unmanaged bees and 81
known Africanized bees, were cut into 4 to 6 pieces and placed in 150 μl of 10%
Chelex® 100 and 5 μl of proteinase K solution (Walsh et al. 1991). The samples were
run in a Bio-Rad Mycycler® for 1 h at 55°C, 15 min at 99°C, 1 min at 37°C, and 15
min at 99°C. Repeated extractions for confirmational purposes were extracted using
the Qiagen® Extraction Kit and purified using the Qiagen® minElute Purification Kit.
The extracted DNA was stored in a -80°C freezer until Polymerase Chain Reaction
A solution of 25 pmoles of E2 (5’-GGCAGAATAAGTGCATTG-3’) and H2
(5’-CAATATCATTGATGACC-3’) primers, 2.5 microliters (μl) of buffer, 2.5 μl at 25
nmoles of each deoxyribonucleotide triphosphate (dNTP), 0.001mg of bovine serum
albumin (BSA), 12.5 μl of deionized water, and 0.5 μl of Taq polymerase were added
to the DNA rendered (Garnery et al. 1993, Delaney 2008)). The amplified DNA was
mixed with blue loading buffer dye from Bioline and run on a 1.4 % agarose gel. The
images were scored and visualized using UV illumination and ethidium bromide. The
restricted fragments were identifiable by base pair length variations (Garnery et al.
The remaining amplified DNA from each sample was digested for 6 h at 37°C
using 0.5 μl of restriction enzyme Dra I, 0.5 (Garnery et al. 1993) μl of buffer, and 4.0
microliters of deionized water per sample. The restricted samples were
electrophoresed on a 7% vertical, acrylamide gel and stained with ethidium bromide
(Delaney 2009). Fragment sizes were estimated and compared to results from Garnery
et al. (1993) in order to assign each sample to a mitochondrial lineage. Haplotypes
were determined based on Restriction Fragment Length Polymorphism (RFLP)
patterns. To confirm patterns, some samples were sequenced. Those samples were
PCR amplified using E2 and H2 primers, then purified using the Qiagen® MinElute
Purification Kit following the standard procedures outlined in the manual before being
sequenced at the Delaware Biotechnology Institute using Applied Biosystems 3730
automatic sequencer. The DNA chromatogram results were viewed and edited using
the program FinchTV and then used a Basic Local Alignment Search Tool (BLAST)
to align the sequences in Genebank to determine exact mitotype.
Twenty microsatellites were used, and then split into three plexes, or
groupings. The loci tested include A88, A113, B124, and AP81, AP66, and A28
(Delaney et al. 2009), as well as A107, A007, HB-THE-03, AC006, and A024, HBTHE-
04, A079, AP043, and UN351 that also show variability and population structure
(Shaibi et al. 2008). Plex 1 consisted of A024, A107, AC006, and HB-THE-03. Plex
2 consisted of A007, A079, AP043, and HB-THE-04. Plex 3 consisted of AP66,
B124, AP81, A88, and A028. Through experimental runs, the maximum effective
number of primers in each plex was found to be five. The primers were combined
based on their base pair lengths in order to avoid overlapping between the primer
ranges. Primers UN351 and A113 did not amplify to a satisfactory degree and were
not used past initial tests.
A comparison of allelic differentiation and private versus common alleles from
distinct sample groupings could be misconstrued by varying sample sizes. A larger
sample size would result in incorrectly larger allelic potential (Kalinowski 2004).
Allelic rarefaction averages sample size for diversity estimation (Kalinowski 2004).
HP-Rare rarefaction software was used for this purpose.
Individual genotypes were assigned to populations using STRUCTURE
(Pritchard et al. 2000). The program uses allele frequencies from nuclear markers to
assign structure, clustering groups into populations based on shared allele frequencies.
The “allelic richness,” or comparison of number of alleles seen in a population
compared to the expected number was performed in HP-rare (Kalinoski 2004).
Genetic differentiation, FST was performed in the software program FSTAT and was
used in estimating genetic differentiation between sampling groups (Kalinowski
2004): unmanaged, managed, known Africanized, and known African. The
unmanaged samples were further separated into three groups: north of North Carolina,
North Carolina, and south of North Carolina.
Genepop was used to determine Hardy-Weinberg Equilibrium. This program
showed the “genetic distinctiveness” between each population and the degree of
change due to natural selection (Kalinowski 2004). If a population has no influence
from intruders into their interactive population, then it is considered to be in
“equilibrium.” The online version performs Hardy Weinberg Exact Test as well as
FST correlations, which were both used. With the Genepop format, it is necessary to
mark two alleles per locus for each individual rather than clustering (Corander et al.
2009). Comparing the private to common alleles will illustrate the interaction, or lack
thereof, between populations.
For microsatellite analysis, samples ran on an Applied Biosystems 3730
automatic sequencer and allele sizes were determined using Genemapper® software
program. Genetic differentiation was assessed using allele frequency data in the
program FSTAT. This technique follows the protocol of previously established
methods (Delaney et al. 2009).
The collection contained limited samples from certain states due to the time
constraints of this project affecting the number of collecting seasons.
Through the geometric wing analysis performed by Dr. David De Jong, several
patterns emerged. The unmanaged and managed samples clustered together,
separately from the representative type of each subspecies (A. m. liguctica, A. m.
mellifera, A.m. carnica and A. m. caucasica) (Figure 2), showing that the bees in the
US are a distinct hybrid. The known Africanized samples separated from each other, however the samples from Florida, Georgia, and Alabama all clustered together but outside the range of Africanized bees (Figure 3). The Africanized bees from Arizona clustered separately from the known Africanized bees from Florida as well as the bees from Africa and Brazil (Figure 3). Based on morphological characters, the bees from Florida may not be fully Africanized.
The unmanaged and managed honey bee colonies were tested and found to
have European mitotypes (Table 7). The Africanized samples from Florida and Belize
contained European mitotypes. The Africanized samples from Arizona had a variety
of mitotypes from the African lineage: A26, A26c, A1e predominantly, and A1. The
Africanized samples from Belize had four European mitotypes that were C1 while the
rest of the mitotypes were mostly A1 and A1e. The samples from Africa and Brazil
had no European mitotypes. They had mitotypes of A4, A47, A26a, and A26c (Table
T Tests were performed in Microsoft Excel to test for significant differences
between the Africanized populations according to mitotypes. The p values for each
comparison were well below the significance level of 0.05, suggesting that the
Africanized samples from Belize, Arizona, Brazil, and Africa had significantly more
mtDNA of African origin compared to the mitotypes of Africanized samples collected from Florida, Alabama, and Georgia. In the comparison of Africanized bees from Florida versus Arizona significant differences were found (p value less than 0.0001,60 degrees of freedom). The comparison of the bees from Florida versus Brazil and Africa was also significant (p value less than 0.0001, 31 degrees of freedom). The p
value for the comparison of Florida versus Belize was 0.001 (96 degrees of freedom).
Even the bees from Arizona, Brazil, Africa, and Belize were significantly different
from each other based on mitotype. The p value for Arizona versus Brazilian and
African samples was 0.000991 (53 degrees of freedom), while the comparison of
Arizona versus Belize was 0.0075038 (118 degrees of freedom). The bees from Brazil
and Africa were different from the bees from Belize (p value = 0.0000509, 89 degrees
The average number of alleles in each population tested ranged from 5.23
±2.12 in Africanized bees from Florida to 7.85 ± 3.25 in unmanaged populations
(Table 2). Allelic richness was the lowest in the Africanized bees from Florida while the Africanized bees from Arizona had the highest allelic richness measured (Table 2).
The expected heterozygosity was very similar with the largest range being between
0.79±0.1 (in samples from Africa and Brazil) and 0.62±0.2 (from the Africanized in
Florida/Alabama/Georgia and Managed populations) (Table 2).
The STRUCTURE output consistently provided two populations: Unmanaged
and managed clustering with the Africanized samples from Florida/Alabama/Georgia
versus Africanized from Arizona, Africanized from Belize, and samples from Africa
and Brazil (Figure 4). STRUCTURE is a program that uses multi-locus data to infer
population structure, assign individuals to populations, and is often used to study
FST stands for fixation index and shows population differentiation due to
structure variation as measured by Single Nucleotide Polymorphisms (SNPs) or
microsatellites. According to the FST results (Table 3), there was a significant
difference between the managed and unmanaged populations. There was also
significance between the African and managed samples. Highly significant FST values
were noted between Africanized bees from Arizona and managed bees and also
between Africanized bees from Arizona and the unmanaged bees. Importantly, the
Africanized samples were not significantly different from each other.
Through the use of Genepop, Hardy Weinberg equilibrium was tested using
Fis, or inbreeding coefficient, estimates to compare each locus for every population. In
each instance, using the Fisher’s Method, the results were highly significant, thus the
populations are not in equilibrium. Allele frequency data, which is the proportion of
alleles compared to the number of genes, can be found in Table 6 in the Appendix.
Currently, Africanized bees are officially diagnosed through morphometric
methods only. The genetics of a honey bee are not often utilized in official
determinations of Africanization. Should we combine morphometric and molecular
data to identify the level of Africanization? Determining the process of Africanization
as either incomplete or absolute can be done with the use of mitochondrial and nuclear
DNA, as was performed in this study, and will serve as a good tool for tracking the
introgression of Africanized genes into commercial and unmanaged honey bee
populations in North America.
Morphometrically, the data shows that there is an American bee, an amalgam
that is distinctive from the originating, European subspecies (Figure 2). The GWV
morphometric study further distinguishes between Africanized bees from Arizona and
those from Africa and Brazil. Interestingly, the samples from
Florida/Alabama/Georgia which were originally diagnosed as Africanized by the
USDA-ID technique were found to have European ancestry using mtDNA markers,
European morphology based on GWV and grouped with managed and unmanaged
honey bee colonies in a population structure analysis using microsatellite markers.
A panmictic population is one in which all forms of recombination are possible
due to a lack of restriction caused by genetics and behavior, and all fertile individuals
are potential partners. We hypothesized that the samples would separate into distinct
populations delineated by lines of latitude. The samples tested in our study are not in
Hardy Weinberg equilibrium, as shown through the use of the Fisher’s test in
Genepop, which is indicative that they are not separate and distinct populations as
would be expected based on geographic collecting locations. Essentially, our study
indicates that bees on the east coast are not in distinct populations as would be
surmised based on collection latitude. Northern bees were not differentiated from
Studies in Europe showed the M lineage, consisting of western and northern
European bees, is genetically much more similar to A, or the African lineage than to C
(eastern Europe) or O lineages (near East and central Asia) (Whitfield et al. 2006).
Our data shows that two populations could be discerned based on microsatellite allele
frequency. The Africanized samples from Arizona and Belize were genetically similar
to the African bees from Africa and Brazil based on allele frequency data on the tested
microsatellites. Furthermore, the unmanaged, managed, and Africanized bees from
Florida/Alabama/Georgia shared enough alleles to be considered one population.
These low levels of introgression can be seen in the Africanized samples collected
from Florida/Alabama/Georgia (STRUCTURE output Figure 4) from the known
African and Africanized populations from Arizona, Belize and Brazil. However, the
Africanized samples from Florida/Alabama/Georgia still grouped with unmanaged and
managed samples collected along the east coast. This could be due to the very recent
Africanization of Florida (2010) or could be due to an incomplete Africanization of
the entire east coast as supplanted by the migratory queen rearing and caged bee trade.
High linkage disequilibrium in A. m. ligustica and A. m. mellifera accounts for the
potential for high genetic variation and recombination (Whitfield et al. 2006). The
opposite is true for A. m. scutellata and A. m, intermissa (Whitfield et al. 2006),
accounting for infiltration of A. m. scutellata into European genetics and not the other
The mtDNA data supplants this conclusion, as the samples of Africanized
bees from Florida/Alabama/Georgia did not exhibit Africanized mitotypes in the
majority of the samples, rather they had European maternal influences, which were also found in the managed and unmanaged samples.
According to Whitfield et al. (2006), New World bees in Brazil were highly
Africanized, yet individuals had alleles from the M lineage. Our data shows bees from
Florida had low levels of Africanization. Mid to late infiltration time showed
substantially, but not exclusively, Africanized genes (Whitfield et al. 2006) which is
substantiated by our data; the Africanized bees from Arizona, one of the earliest states
declared Africanized, were determined to be Africanized based on morphometrics,
mitochondrial DNA, and when tested with microsatellites.
During the process of Africanization in Latin America, bees have historically
had African nuclear and mitochondrial markers become the majority after the first five
to ten years following the invasion (Schneider et al. 2004). This increase of African
nuclear and mitochondrial genes over time is supported by our findings: the samples
from Florida/Alabama/Georgia have European mtDNA and low levels of Africanized
alleles while samples from Arizona consist of mostly African mitotypes and are
accompanied by African nuclear alleles. The onset of the invasion into the
southeastern U.S. as opposed to the desert southwest is supported by the observed
differences in genetic composition between the two sets of samples and is due to the
fact that Arizona was one of the earliest states colonized by Africanized bees due to
the proximity to Central America. It stands to reason that this amount of time allowed
for more complete introgression to occur, making Arizona further along in the
Africanization process. All of the managed and unmanaged honey bees had European
mitotypes (Table 7).
In Texas, the first state to have Africanized bees, it was suggested that the rate
of Africanization was enhanced by the decimation of European feral and managed
colonies by the Varroa destructor mite (Pinto et al. 2005). The Varroa mite created a
selection pressure as Africanized bees survive infestation. It was also suggested that
Texas may be the northernmost range of Africanization possible (Pinto et al. 2005),
while the most recent map of Africanization in the United States shows that as being a
false prediction. This map, however, was constructed using USDA-ID as the
diagnostic tool. If molecular tools were used diagnostically, as in our study, Florida
would not be included. It is possible that the diagnostic techniques (USDA-ID) for
determining Africanized swarms may be sensitive only to phenotype that is not yet
linked to genotype and could find a diagnosis in bees with few genetically African
Genetics should also not be used alone diagnostically. Through the use of
mtDNA testing along with microsatellite data it was shown that the forefront of the
Africanized bee movement is caused by Africanized drones mating with European
queens (Pinto et al. 2005). The resulting hybrid drones were tested and found to be at
a competitive disadvantage (Hall 1991) because of reduced flight speed, eventually
leading to direct competition between European and Africanized queens (Pinto et al.
2005). Our data shows that the Africanized bees from Florida/Alabama/Georgia were morphometrically Africanized only using USDA-ID not the GWV technique, and not necessarily genetically similar enough to the African bee to be considered an Africanized bee.
The Africanized bees from Florida/Alabama/Georgia were different from the
other Africanized samples from other locations using each of the three testing
methods. Using GWV morphometric techniques, the Africanized samples from
Florida/Alabama/Georgia clustered together as being identical to the managed and
unmanaged samples from the north, central, and southern aspects of the eastern
seaboard of the United States and not within the same populations as the Africanized
bees from Arizona, Brazil, or Africa. No clinal gradient in the degree of
Africanization was observed in the unmanaged samples collected along the eastern
seaboard using microsatellite markers. Using mitochondrial testing, the mitotypes
found in the Africanized bees from Florida/Alabama/Georgia proved to be of
European ancestry which differed from the mostly pure African ancestry of the
African and Africanized samples collected from Arizona, Belize, and Brazil. The
difference in mitotypes found in the samples from Arizona, Belize and Africa warrant
further research into possible routes of entry of Africanized colonies.
Using microsatellite analysis to reconstruct actual population dynamics, the
Africanized bees from Florida/Alabama/Georgia clustered into a population with the
unmanaged and managed bees from the east coast while the populations from Arizona,
Belize, Brazil, and Africa clustered together into another separate population. The
lack of definitiveness in the three diagnostic tools leads us to the conclusion that
sensitivity of genetic markers may not be the most useful factor in determining
desirable stock for managing honey bees. Perhaps the best way to evaluate a hive is
by phenotypic traits, such as aggressive tendencies, and other undesirable traits such
as swarming and absconding, and not by genotypic traits. Containing a blend of
markers denotes a level of Africanization but perhaps negative behavior should be the
first line of diagnosis. Excessive stinging and swarming indicates a need for further
tools to be used, but even without the negative diagnoses these traits should be
In conclusion, I hypothesized that unmanaged honey bee colonies have blends
of genes from European and African influences and low levels of Africanization occur
throughout the United States, and can be detected using microsatellite markers. This
blend of genes was made evident by comparing morphometric and mitochondrial test
results. The results from the USDA-ID technique conflicted with the results of the
geometric wing venation, which depicted the Africanized samples from
Florida/Alabama/Georgia as non-Africanized.
For future analysis of our data, more software using Bayesian analysis will
further indicate population structure and degree of admixture. Bayesian Analysis of
Population Structure (BAPS) is one program that shows population structure and
information on introgression. Before processing the samples, we will prepare the data
by forming “group-wise clustering” of the individuals from each population (Corander
et al. 2009). All of the data will be pre-processed through the BAPS program prior to
analysis of genetic information (Corander et al. 2009). Format is the key to successful
analysis. In the future, the reasoning for the difference in sensitivity of the tests
should be explored. Perhaps the diagnoses do not necessitate genotype, as
morphologically and behaviorally the colonies are being confirmed to be Africanized.
The USDA-ID test is performed after a colony is deemed overly aggressive in the
field. With those two delineations, a need for genotypic examination may be
unnecessary. Three of the tools used: geometric wing analysis, mtDNA testing, and microsatellites showed the Africanized bees from Florida, already confirmed
Africanized by USDA-ID, to be incompletely or not at all Africanized.
The probable Africanized drone front is a possibility in the Africanized
counties in Florida, leading to the incomplete genetic introgression. The possibility of
exchanging the morphometric program with a purely molecular diagnostic program is
appealing in that it is more precise, less expensive, and less time consuming, but the
sensitivity may under-estimate the aggressiveness of the hive due to incomplete
Alpatov, V. V. 1929. Biometrical studies on variation and the races of bees.
Quarterly Review of Biology 4: l-58.
Calderone, NW. 2012. Insect Polllinated Crops, Insect Pollinators and US
Agriculture: Trend Analysis of Aggregate Data for the Period of 1992-2009.
PLoS ONE 7(5): e37235. Doi:10.1371/journal.pone.0037235
Clarke, K. E., Oldroyd, B. P., Javier, J., Quezada-Euán, G., & TE Rinderer. 2001.
Origin of honeybees (Apis mellifera L.) from the Yucatan peninsula inferred
from mitochondrial DNA analysis. Molecular ecology, 10(6), 1347–55.
Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11412359
Clarke, K. E., Rinderer, T. E., Franck, P., Quezada-Euán, J. G., & B.P. Oldroyd. 2002.
The Africanization of honeybees (Apis mellifera L.) of the Yucatan: a study of
a massive hybridization event across time. Evolution; international journal of
organic evolution, 56(7), 1462–74. Retrieved from
Corander, J., P. Marttinen, J. Sirén, & J. Tang. 2009. BAPS: Bayesian analysis of
population structure manual. v. 5.3 Last updated 10/28/2009 Åbo Akademi
Delaney, D.A. 2008. Genetic characterization of U.S. honey bee populations. Doctor
of Philosophy thesis at Washington State University.
Delaney, D.A. , M.D. Meixner, N.M. Schiff, & W.S. Sheppard. 2009. Genetic
characterization of commercial honey bee (Hymenoptera: Apidae) populations
in the United States by using mitochondrial and microsatellite markers.
Genetics 102(4): 666-673.
Estoup A., L. Garnery, M. Solignac, & J.M. Cornuet. 1995. Microsatellite variation
in honeybee (Apis mellifera L.) populations: Hierarchical genetic structure and
test of the infinite allele and stepwise mutation models. Genetics 140: 679- 695.
Francoy T.M., P.R. Prado, L.S. Gonçalves, L.D. Costa, & D. De Jong. 2006.
Morphometric differences in a single wing cell can discriminate Apis mellifera
racial types. Apidologie 37: 91-97.
Francoy, T.M., D. Wittman, V. Steinhage, M. Drauschke, S. Müller, D.R. Cunha,
A.M. Nascimento, V.L.C. Figueiredo, Z.L.P. Simoes, D. De Jong, M.C. Arias,
& L.S. Gonçalves. 2009. Morphometric and genetic changes in a population
of Apis mellifera after 34 years of Africanization. Genetics and Molecular
Research 8(2): 709-717.
Francoy T.M., D. Wittmann, M. Drauschke, S. M_ller, V. Steinhage, M.A. Bezerra
Laure, D. De Jong, & L.S. Gonçalves. 2008. Identification of Africanized
honey bees through wing morphometrics: two fast and efficient procedures.
Apidologie 39: 488-494.
Frank P., L. Garnery, A. Louiseau, B.P. Oldroyd, H.R. Hepburn, M. Solignac, & J.M.
Cornuet. 2001. Genetic diversity of the honeybee in Africa: microsatellite
and mitochondrial data. Heredity 86: 420-430.
Garnery L, JM Cornuet, & M Solignac. 1992. Evolutionary history of the honey bee
Apis mellifera inferred from mitochondrial DNA analysis. Molecular Ecology
Garnery, L., M. Solignac, G. Celebrano, & J.M. Cornuet. 1993. A simple test using
restricted PCR-amplified mitochondrial DNA to study the genetic structure of
Apis mellifera L. Experientia 49: 116-1021.
Hall, H. G., & DR Smith. 1991. Distinguishing African and European honeybee
matrilines using amplified mitochondrial DNA. PNAS, 88(10), 4548–52.
Hall, H.G. & K. Muralidharan. 1989. Evidence from mitochondrial DNA that
African honey bees spread as continuous maternal lineages. Nature 339: 211-
Statistics Canada. 2010. Honey highlights. Date modified 12-23-2010, date accessed
Kalinowski, S.T. 2004. Counting alleles with rarefaction: private alleles and
hierarchical sampling designs. Conservation Genetics 5: 539-543.
Kraus, F. B., Franck, P., & R Vandame. 2007. Asymmetric introgression of African
genes in honeybee populations (Apis mellifera L.) in Central Mexico. Heredity,
99(2), 233–40. doi:10.1038/sj.hdy.6800988
McGregor, S.E. 1976. Insect pollination of cultivated crop plant. US Government
Printing Office, Washington, DC.
Needham, G.R., R.E. Page, M. Delfinado-Baker, and C.E. Bowman. 1988.
Africanized honey bees and bee mites. Ellis Horwood Limited.
Neilsen,D.L., P.R. Egbert, G.J. Hunt, E. Guzman-Novoa, S.A. Kinnee, & R.E. Page.
1999. Identification of Africanized honey bees incorporating morphometrics
and improved Polymerase Chain Reaction mitotyping procedure. Annals of
the Entomological Society of America 92(2): 167-174.
Pinto, M. A., WL Rubink, JC Patton, RN Coulson, & JS Johnston. 2005.
Africanization in the United States: replacement of feral European honeybees
(Apis mellifera L.) by an African hybrid swarm. Genetics, 170(4), 1653–65.
Pritchard, J. K., Stephens, M., & Donnelly, P. 2000. Inference of population structure
using multilocus genotype data. Genetics, 155(2), 945–59.
Rinderer, T.E. 1986. Bee genetics and breeding. Academic Press.
Rinderer, T.E., H.A. Sylvester, M.A. Brown, J.D. Villa, D. Pesante, & A.M. Collins.
1986. Field and simplified techniques for identifying Africanized and
European honey bees. Apidologie 17: 33-48.
Rinderer, T.E., S.M. Buco, W.L. Rubink, H.V. Daly, J.A. Stelzer, R.M. Riggio, &
F.C. Baptista. 1993. Morphometric identification of Africanized and European
honey bees using large reference populations. Apidologie 24: 569-585.
Ruttner, F. 1988. Biogeography and taxonomy of honeybees. Springer-Verlag Berlin
Heidelberg publishing, Germany.
Ruttner, F., L. Tassencourt, & J. Louveaux. 1978. Biometrical – statistical analysis of
the geographic variability of Apis mellifera L. Apidologie 9: 363-381.
Sanford, MT. 2006. The Africanized Honey Bee in the Americas: A Biological
Revolution with Human Cultural Implications. American Bee Journal. In 5
Schiff, N.M., W.S. Sheppard, G.M. Loper, & H. Shimanuki. 1994. Genetic diversity
of feral honey-bee (Hymenoptera, Apidae) populations in the southern United
States. Ann. Entomol. Soc. Am. 87: 842-848.
Schneider, S. and G. DeGrandi-Hoffman. 2002. The influence of paternity on virgin
queen success in hybrid colonies of European and African honeybees. Animal
Behaviour 65 (5): 883-892.
Schneider S., T. Deeby, D. Gilley, and G. Degrandi-Hoffman. 2004. Seasonal nest
usurpation of European colonies by African swarms in Arizona, USA. Insectes
Sociaux 51: 359-364.
Shaibi, T., H.M.G. Lattorff, R.F.A. Moritz. 2008. A microsatellite DNA toolkit for
studying population structure in Apis mellifera. Molecular Ecology Resources
Sheppard, W.S. 1989. A history of the introduction of honey bee races into the
United States, part I. American Bee Journal 129(9): 617-619.
Sheppard, W.S. 1997. Subspecies of Apis mellifera pp. 519-533 in: R.A. Morse & K
Flottum (Eds.), Honey Bee Pests, Predators and Diseases, A.I. Root Co.,
Medina, OH, USA.
Sheppard, W. S, & M.D. Meixner. 2003. Apis mellifera pomonella , a new honey bee
subspecies from Central Asia. Apidologie. 34: 367–375.
Vergara C., A. Dietz, & A. Perez. 1989. Usurpation of managed honey bee colonies
in migratory swarms in Tabasco, Mexico. American Bee Journal 129: 824-
Vergara C., A. Dietz, & A. Perez de Leon. 1993. Female parasitism of European
honey bees by Africanized honey bee swarms in Mexico. Journal of
Apicultural Research 32(1): 34-40.
Walsh P.S., D.A. Metzger, & R. Higuchi. 1991. Chelex 100 as a medium for simple
extraction of DNA for PCR-based typing from forensic material.
Biotechniques 10 (4): 506-513.
Whitfield, C. W., Behura, S. K., Berlocher, S. H., Clark, A. G., Johnston, J. S.,
Sheppard, W. S., Smith, D. R., et al. 2006. Thrice out of Africa: ancient and
recent expansions of the honey bee, Apis mellifera. Science (New York, N.Y.),
314(5799), 642–5. doi:10.1126/science.1132772
Wilson, M.L. 1988. The impact of a tropical-evolved honey bee in temperate
climates of North America. Pp. xx-xx. In: G. R. Needham (Ed.). Africanized
honey bees and bee mites. Ellis Horwood Limited publishing co.
Winston, ML. 1992. Killer bees: the Africanized honey bee in the Americas.
Harvard press. Cambridge, MA. 162 pp.