After three years of bitter divorce litigation, Richard Shenkman-Tyler, 60, an advertising executive in South Windsor, Connecticut, took his ex-wife, Nancy Tyler, an attorney, hostage yesterday.  Through a news reporter, he demanded a priest to give her last rights and requested a judge to remarry them.  He told police there were explosives in the house.  Finally after setting the house on fire, Shenkman surrendered to police last night.  Nancy Tyler escaped unharmed.