Eileen Walbert grew up in Virginia and New York. Her parents were from northern cities and had lived in the Philippines, where her mother was a missionary and her father was stationed in the Army. Walbert got married in the early 1940s and lived in New York with her husband, a musician. When they moved to Birmingham in 1946, the Walberts had only heard of the strict segregation in the South. According to Walbert, it was "like moving to Nazi Germany."

At the time, Birmingham was one of the most segregated cities in America. Signs everywhere marked racially segregated water fountains and restrooms. Segregation ordinances made it illegal for African Americans to use the city parks, or for blacks and whites to sit together in any public facility, from restaurants to classrooms. Voter registration tests and scare tactics made it difficult, and sometimes impossible, for African Americans to vote, and hiring discrimination relegated them to menial, low-wage jobs.

Local police and the Ku Klux Klan, a white supremacist organization with a large presence in Birmingham, often enforced segregation with random acts of violence against African Americans. Walbert was shocked to hear that black men were shot in the back, purportedly for resisting arrest or other unsubstantiated charges; and that the Ku Klux Klan castrated a young black man, Judge Aaron, and poured turpentine on his wounds. After the Supreme Court ruled in Brown v. Board of Education (1954), an angry mob beat the Reverend Fred Shuttlesworth when he tried to enroll his children in an all-white school. Shuttlesworth's wife was stabbed. Moved by his courage, Walbert's daughter wrote a letter to Shuttlesworth after the attack.

By the 1960s, the Civil Rights movement had mobilized the black community, but it also generated white resistance. Birmingham remained segregated, and numerous bomb attacks on civil rights activists dubbed the city "Bombingham." In response, the Walberts joined with other whites to form their own civil rights organizations, the Concerned White Citizens of Alabama and the Birmingham Council on Human Relations. Both organizations supported civil rights and worked to improve race relations in Birmingham. They joined other civil rights organizations in demonstrations and marches for civil rights, such as the 1965 Selma march for voting rights, in which police fired tear gas and beat the demonstrators.

The Walberts' support of civil rights not only put them at risk of being attacked during demonstrations, it also made them a target for the Klan and other angry whites. Many of the Walberts' neighbors ostracized them. The Klan burned a cross in their yard as a symbol of white power and intimidation. In addition to harassing the Walberts, the Klan singled out other Concerned White Citizens who supported the Civil Rights movement, publicized their names and addresses in the newspapers, and pressured neighbors and local business to shun them.