Paul Schwarzstein was an ironmonger and metal goods dealer with a business at Freilagergasse 4 in Vienna's 2nd district (today near Vivariumgasse), initially together with his partner Arthur Stemmer and then from 1922 as sole proprietor. He was active in Leopoldstadt on behalf of the Armeninstitut (poorhouse) and until the early 1920s was Armenrat (municipal councillor for the poor). He lived with his wife Leopoldine, née Schönfeld, and their two daughters Julia and Edith Schwarzstein at Matthäusgasse 8 in the 3rd district, where he also had an extensive timepiece collection.

Two weeks after the annexation of Austria to Nazi Germany, the Gestapo seized the timepieces kept in a safe in his apartment and in mid-April 1938 the local tax office imposed a random penalty for non-payment of taxes. A month later, the 65-year-old Schwarzstein suffered a heart attack and died, according to witnesses because of the stress surrounding the Aryanization of his property. The estate and proceeds of the sale of the timepiece collection were to be used to pay the "back taxes", which meant that the widow and daughters were also deprived of their inheritance. Rudolf Kaftan, head of the Uhrenmuseum der Stadt Wien, was particularly interested in the collection and had viewed it in the apartment before it was seized. After drawn-out negotiations with the tax office, mayor Hermann Neubacher, deputy mayor Hanns Blaschke and his adjudant Bartholomäus Schmid, the city of Vienna purchased 135 timepieces for 12,000 Reichsmarks for the Uhrenmuseum and transferred the money to the tax office. Leopoldine Schwarzstein had to move to a collective apartment at Große Mohrengasse 40 in the 2nd district and was deported from there on 5 October 1942 to Maly Trostinec extermination camp, where she was murdered on 9 October 1942. Julia Schwarzstein survived the Nazi era in hiding in Vienna. Edith Schwarzstein, married Fischer, fled to Argentina with her husband Friedrich in April 1939.

Of the 135 timepieces in the collection, 78 were lost in a depot (probably Schloss Thalheim) of the Historisches Museum der Stadt Wien. Fifty-two timepieces were returned to the family in 1949 following a claim, five were given to the Uhrenmuseum, probably as thanks for having stored them, as was often the case immediately after the war. The illegal nature of the acquisition of the five timepieces was determined during the systematic provenance research. Following a decision by the Vienna Restitution Commission, they were returned to Schwarzstein's legal successors in 2006.