On 4 July 1776, American Independence Day, the imperial envoy Giacomo Durazzo in Venice gave Duke Albert von Sachsen-Teschen (1738–1822) and his wife Marie Christine (1742–1798) a collection of engravings he had commissioned two years earlier. This marked the foundation of one of the most important graphic collections in the world. When Albert and Marie Christine settled in Vienna in 1794 as a result of the French Revolution and the loss of the Austrian Netherlands, the art collection was housed in their palace, today's Albertina. From Vienna, the couple purchased first-class works, including the estates of Charles Antoine Prince de Ligne, Cornelis Poos van Amstel and Gottfried Winckler, in this way acquiring works by Michelangelo, Raphael and Leonardo da Vinci as well as major seventeenth- and eighteenth-century Dutch drawings. In exchange for prints, Emperor Francis II (I) gave Albert 500 drawings from the imperial court library, including Dürer's Young Hare and Piece of Turf and works by Raphael and Rembrandt. By the time of his death in 1822, Albert—with the help of his wife's fortune—had increased the collection to around 200,000 prints (including 43,600 from the Italian, 38,700 from the German, 32,200 from the French, 31,000 from the Dutch and 5,100 from the English schools) as well as 14,000 drawings. In his will in 1816, in which he designated his adopted son Archduke Carl (1771–1847) as sole heir, Albert made the collection a family fideicommissum that could not therefore be sold or divided up. At first, the collection, which Carl enlarged to around 16,000 drawings and 220,000 engravings, was available only to artists and scholars and was only opened to the public by Carl's son Archduke Albrecht (1817–1895). In 1899, when the works were already owned by Albrecht's nephew Archduke Friedrich (1895–1936), the first exhibition, with drawings by Albrecht Dürer, was organized in the Albertina.

In 1919, the archducal palace and art collection became the property of the Republic of Austria, and the Albertina merged with the graphic collection of the imperial court library, combining the 18,000 drawings and 220,000 to 230,000 prints belonging to the archduke with over 5,000 (architectural) drawings and 550,000 to 600,000 prints from the former imperial estate. Added to this were around 2,000 architectural drawings from public archives. Alfred Stix, director from 1923, in particular sold duplicates from the print collection to finance new purchases. Stix added around 3,400 drawings in this way. In May 1934, the National Library under Josef Bick took responsibility for the Albertina. Bick was deposed four years later after Austria's annexation to the German Reich on account of his Jewish origins and interned at Dachau concentration camp. The Albertina Graphic Collection was separated from the National Library at the same time, and all “racially” suitable museum staff obliged to swear allegiance to Adolf Hitler. Bick's deputy Anton Reichel took over as temporary director, but was frequently represented by the curator Heinrich Leporini on account of his poor health. Both of them boasted of being the only National Socialists in the Albertina during the years the Nazi Party had been banned. At Leporini's instigation, the art historian Otto Benesch was dismissed in October 1938 because of his marriage to the research assistant Eva Benesch, née Steiner, who was considered a "half-Jew" and had already been dismissed for that reason shortly after the annexation. She was replaced by Magdalena Junk. At the personal request of State Secretary Kajetan Mühlmann, the position of the curator Benno Fleischmann, also dismissed, was taken over by the writer and art historian George Saiko. Purchases in Vienna, Munich, Berlin and Leipzig in particular were designed in these years to increase and close gaps in the nineteenth- and twentieth-century portfolios. Some of the acquisitions were taken from the assets of Jewish collectors. Additions to the Graphic Collection were also made through allocations from the Generalreferat für Kunstförderung, Staatstheater, Museen und Volksbildung (General Office for the Advancement of Art, State Theatre, Museums and Culture) and the Zentralstelle für Denkmalschutz (Central Monument Protection Authority) on the basis of specific requests sent to them. Acquisitions were also made through curator Bernhard  Degenhart's trips to Italy, where he purchased the works of Italian modernists directly from the artists themselves. At Leporini's urging, Mühlmann also suggested the establishment of a miniature collection as a separate entity. Other state collections, such as those of the Kunsthistorisches Museum, the Academy of Fine Arts, the Staatliches Kunstgewerbemuseum in Wien (State Arts and Crafts Museum in Vienna), the Österreichische Galerie and the National Library Portrait Collection, were required to hand over their ivory miniatures to the Albertina. With the assistance of the Vienna Reichsstatthalterei, miniatures were also to be acquired from escaped Jewish collectors.

Even before the outbreak of war, the Albertina, like most other institutions, carried out the instruction to categorize the collections by value and quality and to prepare them for transport to various storage locations. The most important works were originally sent to Gaming, only to be returned shortly afterwards to Vienna, where they were stored in a safe at the headquarters of the Reichsbank in Vienna, now the National Bank. The works were stored in various safes in the city at the instigation of George Saiko, who was responsible for the storage of the Albertina collections. Despite the efforts of Ludwig Berg, who was in charge of storage in the Reichsstatthalterei, Saiko successfully prevented the decentralized storage, except for fifty panels stored in Schloss Ernegg bei Steinakirchen am Forst and a transport of fifty lots to Lauffen near Bad Ischl at the end of 1944. To be able to put on exhibitions despite the absence of most of the museum's inventory, an exhibition series The Unknown Albertina was inaugurated during the war featuring facsimiles. The works of living artists were a further focus. While the collection remained safe from air raids, the north-eastern wing of Albertina palace was destroyed by a bomb on 12 March 1945.

After the war, Leporini, a staunch Nazi who had taken over as director after Reichel's death in February 1945, was declared "acceptable" under the Prohibition Act and retired. After just a few days' intermezzo with Saiko as director, Karl Garzarolli-Thurnlackh took over the position. Under him, the Albertina did not open its doors to the public again until 1 June 1946. Garzarolli also carried out a major exchange with Carnegie Hall, New York, which received duplicates from the Albertina. After his return from exile in the USA in 1947, Otto Benesch was reinstated as director of the Albertina. Although himself a victim of Nazi persecution, he was involved in stopping the export of restituted works from art collections expropriated between 1938 and 1945, which were donated to the Albertina or purchased at low cost.

Whereas around 4,000 drawings and 3,000 prints had been acquired since May 1934, the Albertina collection under Benesch acquired an additional 3,600 drawings and 2,400 prints. Today it has around 40,000 drawings and watercolours, 25,000 architectural drawings held in a department of their own, and at least 900,000 prints.

Since the Art Restitution Act entered into force in 1998, many works from the Albertina acquired from 1938 (1933) until the present have been returned. They include works by prominent collectors who appear in recommendations for restitution from other federal museums, as well as persons specializing solely in drawings or prints. The Art Restitution Advisory Board made recommendations regarding the following collections: Alexander Beer, Gaston Albert Belf, Michael Berolzheimer, Rudolf Bittmann, Josef Blauhorn, Ferdinand Bloch-Bauer, Betty Blum, Oscar Bondy, Otto Brill, Caroline Czeczowiczka, Valerie Eisler, Arthur Feldmann, Adella Feuer, Julius Freund, David Goldmann, Moriz Grünebaum, Fritz Grünbaum, Henri and Pauline Grünzweig, Rudolf Gutmann, Valerie Heissfeld, Rudolf Hirschenhauser, Bruno Jellinek, Hubert Jung, Eva Kantor, Siegfried Kantor, Maximilian Kellner, Gustav Kirstein, Hermann Kolisch, Hans Körbel, Hans Lothar Körner, Gottlieb Kraus, Moritz Kuffner, Siegfried Lämmle, Erich Lederer, Hans Leinkauf, Julius Mannaberg, Karl Mayländer, Alfred and Fritz Menzel, Edith Oser-Braun, Adalbert Parlagi, Adele Pächter, Ignatz Pick, Albert Pollak, Oskar Reichel, Armin Reichmann, Heinrich Rothberger, Clarice and Louis Rothschild, Marianne Schmidl, Gertrude Schüller, Luise Simon, Valentine Springer, Emerich Ullmann, Richard Weinstock, Josefine Winter and Mary Wooster.