Children Joseph and Fanny had nine children:
Francis Robinson (1863–1938)
Fanny de Brissac Eliza (1865–1870)
Mary Selina Margaret (1867–1939)
Anne Isabella Emma (1868–1966)
Fanny Josephine (1873–1964)
Harriot Edith de Brissac (1875–1949)
Sophia Maud Emily (1876–1979)
Joseph Bryan William (1879–1941)
Charles Edward Feild (b & d 1880) *
* [Note: Joe's last child was named after the Bishop of Newfoundland, Edward Feild, often misspelt Field.]
The family hoped that Joe, the eldest son, would follow his father into the wine trade after completing his schooling with his uncle, Dr A. B. Evans.
He did not show any inclination or aptitude for trade, however, and went to study theology at St Augustine's College, Canterbury.
He was sent to Newfoundland (where there was a settlement of Portuguese speakers) was ordained there and served in the cathedral at St John's, the mission at Portugal Cove, as vice-principal of the theological college in St John's and as headmaster of the Church of England School there.
In 1862 he married Fanny Harriot, daughter of Sir Bryan Robinson, a politician and judge in Newfoundland. They had nine children, two of whom died young. Their eldest son, also a clergyman, became in turn Bishop of Grahamstown and Archbishop of Cape Town, in South Africa.
After a long line of Josephs in this branch of the Phelps family, Joe's second son was the last to bear this name, and in fact the Phelps surname is due to die out when the last male Phelps dies unmarried.
In 1884 Joe and Fanny moved to England, Joe taking up various clerical appointments to make ends meet, and eventually retired with their unmarried daughters, to Rivermead in Iffley, Oxon, where a large family gathering was held to mark their Golden Wedding in 1912.
Joe suffered poor eyesight in old age but took a keen interest in family history, updating the book which his sister Fanny had made for him and gathering much data about his wife's Robinson forebears. He sent his sister Harriet a book called Ancestral Tablets, which her son Godfrey kept up to date after her death.
Joseph died at Rivermead, aged 93.